HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Thu, 28 Nov 2024 05:50:36 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"674804ac-53c5" expires: Tue, 30 Dec 2025 02:25:24 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: 8CF1:36A0B4:979FC7:AA577B:695335BC accept-ranges: bytes age: 0 date: Tue, 30 Dec 2025 02:15:24 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210063-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1767060924.413668,VS0,VE217 vary: Accept-Encoding x-fastly-request-id: dfe817cfb951469b5edf2a6da651a8435072f96b content-length: 5069

CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation

Jun Wang^1*, Yuzhe Qin^1*, Kaiming Kuang¹, Yigit Korkmaz², Akhilan Gurumoorthy¹, Hao Su¹, Xiaolong Wang¹

¹UC San Diego; ²USC
CVPR 2024
^*Indicates Equal Contribution

Paper Code Video

We introduce CyberDemo, a novel approach to robotic imitation learning that leverages simulated human demonstrations for real-world tasks.

Abstract

By incorporating extensive data augmentation in a simulated environment, CyberDemo outperforms traditional in-domain real-world demonstrations when transferred to the real world, handling diverse physical and visual conditions. Regardless of its affordability and convenience in data collection, CyberDemo outperforms baseline methods in terms of success rates across various tasks and exhibits generalizability with previously unseen objects. For example, it can rotate novel tetra-valve and penta-valve, despite human demonstrations only involving tri-valves. Our research demonstrates the significant potential of simulated human demonstrations for real-world dexterous manipulation tasks.

                Motivation

                Video Presentation

Method

CyberDemo Pipeline. First, we collect both simulated and real demonstrations via vision-based teleoperation. Following this, we train the policy on simulated data, incorporating the proposed data augmentation techniques. During training, we apply automatic curriculum learning, which incrementally enhances the randomness scale based on task performance. Finally, the policy is fine-tuned with a few real demos before being deployed to the real world.

Real World Deployments

Pouring Task

In Domain

Out of Position

Out of Position + Random Disco Light

Rotating Task

In Domain

Diverse Objects: Tetra Valve

Diverse Objects: Penta Valve

Random Light

Pick And Place Task

Novel View + Out of Position

Robust Policy

BibTeX

@inproceedings{wang2024cyberdemo,
        title={CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation},
        author={Wang, Jun and Qin, Yuzhe and Kuang, Kaiming and Korkmaz, Yigit and Gurumoorthy, Akhilan and Su, Hao and Wang, Xiaolong},
        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
        pages={17952--17963},
        year={2024}
      }

Acknowledgement

We gratefully acknowledge support from the Technology Innovation Program (20018112, Development of autonomous manipulation and gripping technology using imitation learning based on visual and tactile sensing) funded by the Ministry of Trade, Industry & Energy (MOTIE), Korea.