PyTorch implementation of "On the Utility of 3D Hand Poses for Action Recognition", ECCV 2024.

Poses

Download AssemblyPoses.zip from here. For access, follow the instructions from Assembly101 official repo.

$ unzip AssemblyPoses.zip -d /<path_to_your_data_dir>/assembly101_poses/

RGB

For experiments with RGB, we provide DINOv2 features (at 30fps) for view 4 (Camera ID: C10119_rgb) of Assembly101 here.

Alternatively, TSM features for any of the views can be downloaded from the same directory.

Frame-wise TSM or DINOv2 features are stored as lmdb files.

Annotations

Download the CSV files from fine-grained annotations and put them into <path_to_your_data_dir>/Assembly101_FG_annotations/.

Details about the annotations can be found here.

Create a conda environment with the necessary packages.

$ conda env create -f package_list.yml -n handformer_env

The parameters and data directories for preprocessing are placed in Preprocess/config.py. Follow the instructions there to modify and run the following:

$ python Preprocess/1_get_clip_data.py
$ python Preprocess/2_get_final_data.py

All the parameters and data paths for training are defined and explained in train_config.yaml. The parameters are populated with default values. Modify paths (and parameters, if necessary) and run-

$ python HandFormer/train_model.py

Both Pose-only and Pose+RGB variants can be found in the configuration file. If RGB is used, rgb_feature_source needs to be specified. Additionally, update data paths in HandFormer/feeders/feeder.py (within lines 80-115) to refer to the downloaded lmdb files of TSM or DINOv2 features.

To obtain test scores, simply put additional placeholder columns in test.csv to match train.csv. Set parameters in test_config.yaml and run-

$ python HandFormer/test_model.py

Prepare appropriate output file from the saved scores to submit to the evaluation platforms (e.g., [Assembly101][H2O].)

This repository is based on

• Assembly101
• MS-G3D
• ISTA-Net
• HTT

Thanks for their great work!

Please cite this work if you find it useful:

@inproceedings{shamil2025utility,
  title={On the Utility of 3D Hand Poses for Action Recognition},
  author={Shamil, Md Salman and Chatterjee, Dibyadip and Sener, Fadime and Ma, Shugao and Yao, Angela},
  booktitle={European Conference on Computer Vision},
  pages={436--454},
  year={2025},
  organization={Springer}
}