Bi-Adapt: Few-shot Bimanual Adaptation for Novel Categories of 3D Objects via Semantic Correspondence

Teaser

Pipeline

(Left) We collect quantities of data in simulator, and train the Action Learning Network to build our supporting set. The corresponding affordance and point-level action of training categories serve as our prior knowledge. (Middle) When facing an unseen object from novel categories and a task, we retrieve the familiar ones from our training categories within the same task. We leverage the powerful semantic correspondence capability in the diffusion model to achieve cross-category affordance generalization. Particularly, we often get multiple contact-point pair candidates. (Lower Right) Then, we sample the affordance candidates and action directions proposed by pre-trained Action Learning Network to execute. After few-shot learning, our network is fine-tuned by the interaction results to adapt the novel categories. (Upper Right)Finally, our fine-tuned framework can facilitate manipulating unseen instances from novel categories with better performance.

More Visuals

Cross-category affordance generalization results

In each group of figures from left to right, we present a source image as reference (★ represents the first gripper’s contact point and ★ represents the second gripper’s contact point), and a target image with corresponding contact point inferred by diffusion model. The degree of highlighting in the target image represents the degree of correspondence.

Adaptation Results

Visualization of manipulation results

In each row, from left to right, we respectively show the process of task by ar- rows above. The blue block represents action learning on training categories, building supporting set. The green block shows the process of contact points mapping to novel categories. After few- shot learning on novel categories (yellow block), our finetuned model could select best contact point pairs and propose suitable actions for novel categories in pink block.

Visualization of affordance comparison

Real Experiment Setup

(a) Robot Setup

(b) Objects

(a) We use two UFactory xArm6 for manipulation and two RealSense D435 cameras to cap- ture RGBD observations.
(b) We test on a diverse set of objects.

Visualization of real-world experiments

Each row represents a different task: (top) unfolding, (middle) opening, and (bottom) uncapping.
For each row, from left to right: (a) the target object from unseen categories; (b) contact points of training categories; (c) contact points mapped on novel categories; (d) the initial pose of the grippers; (e) the final result after manipulation.