| CARVIEW |
Bimanual Dexterity for Complex Tasks
Jiahui Yang Mohan Kumar Srirama Ray Liu Haoyu Xiong
Russell Mendonca† Deepak Pathak†
Carnegie Mellon University
*Equal contribution. †Equal Advising.
CoRL 2024Abstract
To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms ($16k USD).
Autonomous Results
Serving
Drill Operating
Pouring in the wild
Plate Pickup
Pouring
Wire Winding
Teleoperation Results
Table Rearragement
Card Pickup
Dining Rearrangement
Chopstick Pickup
Grape Plucking
Pour Chips
Scoop and Pour
Stack Bowl
Stack Cups
Stack Plate
Insert Brush
Cut Cucumber
Peel Cucumber
Pouring
Mix with Spatula
Spray and Clean with Brush
Whisking
Use Stapler
Screwdriver Pickup
Tool Cleanup
Use Drill
Hammering Dough
Rearrange and Insert Soldering Iron
Wire Winding
Hang Shirt
Fold Shirt
Teleoperation with Operator
Stack Plate
Stack Bowl
Grape Plucking
Mobile Teleoperation
Operator View of Box Transport
Clear Trash
Box Transport
Push Chair
LEAP Hand v1 Autonomous
Pouring
Pringles (Dynamic)
Stack Cup
BibTeX
@inproceedings{shaw2024bimanual,
title={Bimanual Dexterity for Complex Tasks},
author={Shaw, Kenneth and Li, Yulong and Yang, Jiahui and Srirama, Mohan Kumar and Liu, Ray and Xiong, Haoyu and Mendonca, Russell and Pathak, Deepak},
booktitle={8th Annual Conference on Robot Learning},
year={2024}
}Acknowledgements
We thank Ankur Handa, Arthur Allshire, Toru Lin and Nancy Pollard for discussions about the paper. We also thank Maarten Witteveen and Sarah Shaban at Manus Meta for their assistance on their gloves. We thank Diya Dinesh and Lukas Kebuladze for helping with teleoperation. This work is supported in part by DARPA Machine Commonsense grant, AFOSR FA9550-23-1-0747, ONR N00014-22-1-2096, AIST CMU grant and Google Research Award.