The official codebase for running the experiments described in the AVDC paper. You can find codebase for training video policies here.

Learning to Act from Actionless Videos through Dense Correspondences
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, Joshua B. Tenenbaum
website | paper | arXiv | video policy

@article{Ko2023Learning,
  title={{Learning to Act from Actionless Videos through Dense Correspondences}},
  author={Ko, Po-Chen and Mao, Jiayuan and Du, Yilun and Sun, Shao-Hua and Tenenbaum, Joshua B},
  journal={arXiv:2310.08576},
  year={2023},
}

We recommend to create a new environment with pytorch installed using conda.

conda create -n avdc_exp python=3.9
conda activate avdc_exp
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

Next, clone the repository and install the requirements

git clone https://github.com/flow-diffusion/AVDC_experiments
cd AVDC_experiments
pip install -r requirements.txt

We provide the checkpoints used in our main experiments. You can download them using download.sh, for example:

bash download.sh metaworld
# bash download.sh metaworld-DA
# bash download.sh ithor 

First, cd into the experiment directory.

cd experiment

To run the full AVDC on Meta-World, run the following command:

# make sure you have the checkpoint ../ckpts/metaworld/model-24.pt
bash benchmark_mw.sh 0
# the argument 0 is the GPU id, you can change it to other GPU id if you wish

We have provided also provided another checkpoint trained with simple random-shift data augmentation. Specifically we first center cropped the image to 160x160 from the original 320x240 image and then random-crop an 128x128 image from it. We found slightly improved performance with this simple augmentation.

To run the full AVDC on Meta-World with this checkpoint, run the following command:

# make sure you have the checkpoint ../ckpts/metaworld_DA/model-24.pt
bash benchmark_mw_DA.sh 0

To run the full AVDC on iTHOR, run the following command:

# make sure you have the checkpoint ../ckpts/ithor/model-16.pt
bash benchmark_thor.sh 0

This codebase is modified from the following repositories:
unimatch
imagen-pytorch
guided-diffusion