This repository contains the codebase for our paper:

ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis
Yu Fang, Yue Yang, Xinghao Zhu, Kaiyuan Zheng, Gedas Bertasius, Daniel Szafir, Mingyu Ding

Project Page | arXiv

We use ZED SDK 4.1 to export videos and metadata from the DROID dataset. Please follow the official installation guide.

conda create -n zed python=3.10
conda activate zed
./path/to/runfile.run

Please enable AI modules during installation, and we recommend answering 'y' to all prompts.

Then install other required Python packages:

pip install open3d h5py trimesh pillow opencv-python

For real-world trajectory replay, we use Isaac Sim 4.1 (we recommend binary installation) and Isaac Lab 1.1.0 (already included in this repo). We recommend a compatibility check to make sure your device meet the Isaac Sim requirements.

Set up the conda environment and install dependencies:

./isaaclab.sh --conda
conda activate isaaclab
sudo apt install cmake build-essential
./isaaclab.sh --install
pip install h5py scipy

Please refer to the official installation guide for more details. Please verify the installation to make sure it’s successful.

Install Grounded-SAM-2 and ProPainter following their official instructions in separate conda environments ("groundedsam2" and "propainter"). Set your path to Grounded-SAM-2 in processing/droid_plane_analysis.py and inpainting/object_seg.py, and your path to ProPainter in inpainting/object_rm.py.

We support three datasets including DROID, BridgeData V2, and our dataset. Please download the raw versions for DROID and BridgeData V2.

For the DROID dataset, download the raw version and unzip it to data/droid_raw, then:

# export videos and metadata
conda activate zed
python processing/droid_data_export.py
# estimate table height
conda activate groundedsam2
python processing/droid_plane_analysis.py

For BridgeData V2 and our dataset, camera parameters and table height are fixed and included in the code.

For our dataset, download and unzip it to data/rebot_data and export videos:

conda activate zed
python processing/rebot_data_export.py

Eventually, the episode folders should be organized under data/droid, data/bridge, and data/rebot respectively. The folder structure should look like this:

data
├── droid
│   ├── episode_1
│   ├── episode_2
│   ├── ...
├── bridge
│   ├── episode_1
│   ├── episode_2
│   ├── ...
├── rebot
│   ├── episode_1
│   ├── episode_2
│   ├── ...

We use conda environment “isaaclab” for this component. Please use absolute path in arguments for running Isaac Lab.

conda activate isaaclab

Testrun for estimating the object positions. Set the dataset name ("droid", "bridge", or "rebot") and the dataset directory in the arguments.

python trajectory_replay/script_replay.py \
    --dataset droid \ 
    --dataset_dir /path/to/rebot/data/droid/ \ 
    --testrun

Replay the trajectory. Choose the object and container from the list in isaaclab/trajectory_replay/objects_*.json, and specify their names respectively in --object and --container.

python trajectory_replay/script_replay.py \
    --dataset droid \
    --dataset_dir /path/to/rebot/data/droid/ \
    --save \
    --object carrot \
    --container plate

After trajectory replay, validate whether the replayed trajectories are successful. Specify the episode name in the format "object-container" (e.g., "carrot-plate") in the argument --episode.

python trajectory_replay/validate.py \
    --dataset droid \
    --dataset_dir ../data/droid \
    --episode carrot-plate

Check if the object is upright. Set the upright direction of the object in the json file, and use --check_rotation to verify whether the object remains upright after manipulation.

Custom 3D Objects. We provide several sample objects under isaaclab/source/extensions/omni.isaac.lab_assets/data/Objects/. To add custom 3D objects to the simulation, convert obj file to usd format and register the object in isaaclab/trajectory_replay/objects_*.json. We recommend normalizing the object scale before running the command below.

cd isaaclab
./isaaclab.sh -p \
    source/standalone/tools/convert_mesh.py \
    path/to/object_name.obj \
	source/extensions/omni.isaac.lab_assets/data/Objects/object_name/object_name.usd \
    --collision-approximation convexDecomposition \
    --mass 1.0

Custom Robots. We provide the usd files of Franka Panda with Robotiq-2F85 gripper and WidowX. To add custom robots, put the usd file under isaaclab/source/extensions/omni.isaac.lab_assets/data/Robots/ and register the robot under isaaclab/source/extensions/omni.isaac.lab_assets/omni/isaac/lab_assets. We recommend using the Physics Inspector in Isaac Sim to check if it works well.

Tip: If the robot videos are recorded in the same scene with a fixed camera (e.g., BridgeData V2 and our dataset), feel free to skip this step and use an empty or manually inpainted background. We provide example empty backgrounds for BridgeData V2 and our dataset in the synthesis folder.

Segment the robot and the manipulated object in the video using Grounded-SAM-2:

conda activate groundedsam2
python inpainting/object_seg.py \
    --dataset droid \
    --dataset_dir data/droid/

Use ProPainter for video inpainting:

conda activate propainter
python inpainting/object_rm.py \
    --dataset droid \
    --dataset_dir data/droid/

python synthesis/synthesize.py \
    --dataset bridge \
    --dataset_dir ../data/bridge \
    --episode carrot-plate

We recommend converting synthetic videos into RLDS format for compatibility with most VLA training/finetuning pipelines. Please follow the instructions in this repo. We provide example builders in the rlds_builder folder, please update the paths to your folders in the _split_generators function.

We use Octo and OpenVLA for our experiments. Please follow the official installation guides and finetuning instructions. For Octo, we recommend using the 1.0 version following the installation guide in SimplerEnv.

We use SimplerEnv for evaluation in simulated environments. Please check this repo for installation and usage instructions. We provide our evaluation scripts in the evaluation folder.

If you find our work useful, please cite our paper:

@article{fang2025rebot,
  title={ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis},
  author={Fang, Yu and Yang, Yue and Zhu, Xinghao and Zheng, Kaiyuan and Bertasius, Gedas and Szafir, Daniel and Ding, Mingyu},
  journal={arXiv preprint arXiv:2503.14526},
  year={2025}
}