Zeqi Xiao  Tai Wang  Jingbo Wang  Jinkun Cao  Wenwei Zhang  Bo Dai  Dahua Lin  Jiangmiao Pang* 
Shanghai AI Laboratory Nanyang Technological University Carnegie Mellon University

• For reproducing the demo, please refer to blender_for_UniHSI
• [2024-04] The data is released.
• [2024-03] The code is released.
• [2024-01] UniHSI is accepted as ICLR 2024 spotlight. Thanks for the recognition!
• [2023-09] We release the paper of UniHSI. Please check the 👉 webpage 👈 and view our demos! 🎇;

Download Isaac Gym from the website, then follow the installation instructions.

Once Isaac Gym is installed, install the external dependencies for this repo:

pip install -r requirement.txt

1. Download PartNet and ShapeNet V2.

2. Save them in the following formation

data/
├── partnet_origin
│   ├── obj_id1
│   ├── obj_id2
│   ├── ...
├── shapenet_origin
│   ├── class_id1
│   │    ├── obj_id1
│   │    ├── ...
│   ├── class_id2
│   │    ├── obj_id1
│   │    ├── ...
│   ├── ...

3. Extract the objects used in sceneplan by

python cp_partnet_train.py
python cp_partnet_test.py

1. Download ScanNet.

2. Save them in the following formation

data/
├── scan_origin
│   ├── scans
│   │   ├── scans_1
│   │   ├── scans_2
│   │   ├── ...

3. Extract the objects used in sceneplan by

python cp_scannet_test.py

We select and process motion clips from SAMP and CIRCLE.

We adopt step-by-step training.

sh train_partnet_simple.sh
sh train_partnet_mid.sh
sh train_partnet_hard.sh

sh demo_scannet.sh

sh test_partnet_simple.sh
sh test_partnet_mid.sh
sh test_partnet_hard.sh
sh test_scannet_simple.sh
sh test_scannet_mid.sh
sh test_scannet_hard.sh

The results will be saved in the "output" folder.

• There will be ~10% variance due to randomness in sampling.

If you find our work helpful, please cite:

@inproceedings{
  xiao2024unified,
  title={Unified Human-Scene Interaction via Prompted Chain-of-Contacts},
  author={Zeqi Xiao and Tai Wang and Jingbo Wang and Jinkun Cao and Wenwei Zhang and Bo Dai and Dahua Lin and Jiangmiao Pang},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/forum?id=1vCnDyQkjg}
}

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

• ASE: Our codebase is built upon the AMP implementation in ASE.
• PartNet and ShapeNet.: We use objects from PartNet for training and evaluation.
• ScanNet: We use scenarios from ScanNet for evaluation.
• SAMP: We use motion clips from SAMP for training.
• CIRCLE: We use motion clips from CIRCLE for training.