Official Pytorch implementation of Sim2Seg (paper, site) by

John So*, Amber Xie*, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali-akbar Agha-mohammadi, Pieter Abbeel, and Stephen James.

*equal contribution

Install the following libraries:

sudo apt update
sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3

Install dependencies:

conda env create -f conda_env.yml
conda activate sim2seg

We release a sample Unity driving environment. We use the Free Stylized Nature Environment and the free Realistic Buggy Kit. To try running Sim2Seg on this basic environment, download the files from this link.

In our paper, we use a set of diverse and more realistic environments. You can purchase them on the Unity Asset store:

• Meadow: Dyamic Nature Meadow Environment
• Landscapes: Real Landscapes Valley Forest
• Canyon: Castle Valley Collection 1

To download a Sim2Seg segmentation model for training in the segmentation space, visit this link.

To run headlessly, follow instructions at RLBench.

Train the agent:

python3 train.py experiment_folder=first_folder experiment_name=first_experiment

To train on multiple environments set use_switch_every=True and set eval_task_names and train_task_names to the train and evaluation environments. Note that we switch the Sim2Seg segmentation model per each environment to avoid distribution shifts between environments. For our real-world deployment, we use a separate Sim2Seg segmentation model trained on all environments. We currently only provide a sample Sim2Seg model, so all models are the same.

To train without Sim2Seg, set use_s2s=False.

To train headlessly, refer here for headless rendering w/ VirtualGL!

If you use this code for your research, please cite our paper:

@inproceedings{
so2022simtoreal,
title={Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data},
author={John So and Amber Xie and Jeffrey Edlund and Rohan Thakker and Sunggoo Jung and Ali-akbar Agha-mohammadi and Pieter Abbeel and Stephen James},
booktitle={6th Annual Conference on Robot Learning},
year={2022},
url={https://openreview.net/forum?id=eyxfGTFZbNQ}
}

We build off of the following repositories / papers:

• DrQv2
• pix2pix

Sim2Seg is licensed under the MIT license.