articulate-anything Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

In-the-wild reconstructions

In this demo, we visualize the articulation results for in-the-wild videos. These videos were casually captured on an iPhone (e.g., titled angles) in cluttered environments, showing Articulate-Anything's ability to handle diverse, complex, and realistic object affordances.
Note that Articulate-Anything can resolve ambiguous affordance by leveraging grounded video inputs. For example, double-hung windows can potentially either slide or tip to open. This affordance is not clear from static image. When Articulate-Anything is shown a demonstration video, it can accurately produce the desired model in simulation.

Below, we show the Python output generated by our system and the corresponding digital twin rendered in Sapien simulator.

Robotics Application

Without articulation, objects can only afford trivial interaction such as pick and place. We show that articulate Articulate-Anything's output can be used to create digital twins in simulation for robotic training of finer-grained manipulation skills, from closing a toilet lid to closing a cabinet drawer. These skills can be then transferred to a real-world robotic system in a zeroshot manner. Here, we visualize the policies trained using PPO in a simulator using Robosuite library for 2 million episodes over 3 random seeds per task.

Mesh Generation

Currently, Articulate-Anything employs a mesh retrieval mechanism to exploit existing 3D datasets. Open repositories such as Objaverse contain more than 10 million static objects. Our system can potentially bring those objects to life via articulation. However, as stated, a promising future direction to enable the generation of even more customized assets is to leverage large-scale mesh generation models. Below, we show a few examples of meshes generated by Rodin using in-the-wild images. The static meshes are then articulated by Articulate-Anything.

Quantitative Results

Prior works such as URDFormer and Real2code rely on impoverished inputs such as cropped images or text bounding box coordinates. Impoverished inputs also mean that their articulation had to be done on an open loop.
On the standard PartNet-Mobility dataset, Articulate-Anything, leveraging grounded video inputs and closed-loop actor-critic system, substantially outperforms these prior works in automatic articulation, setting a new bar in state-of-the-art performance.

In an ablation experiment, we found that indeed richer and more grounded modalities such as videos enable higher articulation accuracy in our own system.

We also evaluate the effect of in-context examples on success rates for link placement and joint prediction tasks. The results show that providing more in-context examples significantly improves performance.

Finally, by leveraging a visual critic, Articulate-Anything can self-evaluate its predictions and self-improve over subsequent iterations.