| CARVIEW |
Select Language
HTTP/2 200
server: GitHub.com
content-type: text/html; charset=utf-8
last-modified: Thu, 24 Jul 2025 00:23:42 GMT
access-control-allow-origin: *
etag: W/"68817d0e-3234"
expires: Wed, 31 Dec 2025 12:51:41 GMT
cache-control: max-age=600
content-encoding: gzip
x-proxy-cache: MISS
x-github-request-id: 1090:123DE:B344A3:C9A046:69551A05
accept-ranges: bytes
age: 0
date: Wed, 31 Dec 2025 12:41:41 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210051-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1767184901.340277,VS0,VE216
vary: Accept-Encoding
x-fastly-request-id: bd881bf68f5a47a63f823c67820e5a6d1136fe11
content-length: 3296
Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
Jad Abou-Chakra1,
Lingfeng Sun1,
Krishan Rana2,
Brandon May 1
,
Karl Schmeckpeper 1
,
Maria Vittoria Mnniti1,
Laura Herlant1
1Robotics and AI Institute
2Queensland University of Technology
The real-is-sim framework operating with an always-in-the-loop simulator. The learnt PushT policy acts on the simulator and the simulator synchronizes itself with the real world. The real robot mimics the simulated robot.
Abstract
We introduce real-is-sim, a new approach to integrating simulation into behavior cloning pipelines. In contrast to real-only methods, which lack the ability to safely test policies before deployment, and sim-to-real methods, which require complex adaptation to cross the sim-to-real gap, our framework allows policies to seamlessly switch between running on real hardware and running in parallelized virtual environments. At the center of real-is-sim is a dynamic digital twin, powered by the Embodied Gaussian simulator, that synchronizes with the real world at 60Hz. This twin acts as a mediator between the behavior cloning policy and the real robot. Policies are trained using representations derived from \emph{simulator} states and always act on the simulated robot, never the real one. During deployment, the real robot simply follows the simulated robot’s joint states, and the simulation is continuously corrected with real world measurements. This setup, where the simulator drives all policy execution and maintains real-time synchronization with the physical world, shifts the responsibility of crossing the sim-to-real gap to the digital twin's synchronization mechanisms, instead of the policy itself. We demonstrate real-is-sim on a long-horizon manipulation task (PushT), showing that virtual evaluations are consistent with real-world results. We further show how real-world data can be augmented with virtual rollouts and compare to policies trained on different representations derived from the simulator state including object poses and rendered images from both static and robot-mounted cameras. Our results highlight the flexibility of the real-is-sim framework across training, evaluation, and deployment stages.
Framework
The real-is-sim framework seamlessly transitions between online and offline modes because it uses an always-in-the-loop simulator as a mediator.
Learnt Policies
Policies learnt from different representations extracted from the simulator state.
Online Mode
Our system operating in online mode with Embodied Gaussians acting as a mediator between the policy and the real world.
Offline Mode
Our system operating in offline mode evaluating an imitation learning policy. Offline mode can render virtual cameras for multiple scenes at the same time and evaluate vision based policies as well as state based policies.
Gripper Virtual Camera
An imitation learning policy trained on a virtually rendered image from a virtual camera mounted on the end effector.
Static Virtual Camera
An imitation learning policy trained on a virtually rendered image from a static virtual camera.
Interesting Exploration Behaviour
The imitation learning policy explores the space to find the T-Block when the T-Block is not in view of its virtual gripper camera.
Desynchronization
The imitation learning policy creates actions that actively prevent the visual correction mechanism from resynchronizing the simulator.
Related Links
Embodied Gaussians is the underlying simulator that powers real-is-sim.