• 2022.01: Add the trained model download link of google driver.

• Prerequisites
• Data Preparation
• Uodates
• Pretrained Models
  • Kinetics-400
  • Hmdb51 && UCF101
• Testing
• Training
• Contributors
• Citing_ActionClip
• Acknowledgments

The code is built with following libraries:

• PyTorch >= 1.8
• wandb
• RandAugment
• pprint
• tqdm
• dotmap
• yaml
• csv

For video data pre-processing, you may need ffmpeg.

More detail information about libraries see INSTALL.md.

We need to first extract videos into frames for fast reading. Please refer to TSN repo for the detailed guide of data pre-processing. We have successfully trained on Kinetics, UCF101, HMDB51, Charades.

• We now support single crop validation(including zero-shot) on Kinetics-400, UCF101 and HMDB51. The pretrained models see MODEL_ZOO.md for more information.
• we now support the model-training on Kinetics-400, UCF101 and HMDB51 on 8, 16 and 32 frames. The model-training configs see configs/README.md for more information.
• We now support the model-training on your own datasets. The detail information see configs/README.md.

Training video models is computationally expensive. Here we provide some of the pretrained models. We provide a large set of trained models in the ActionCLIP MODEL_ZOO.md.

We experiment ActionCLIP with different backbones(we choose Transf as our final visual prompt since it obtains the best results) and input frames configurations on k400. Here is a list of pre-trained models that we provide (see Table 6 of the paper). *Note that we show the 8-frame ViT-B/32 training log file in ViT32_8F_K400.log.

On HMDB51 and UCF101 datasets, the accuracy(k400 pretrained) is reported under the accurate setting.

To test the downloaded pretrained models on Kinetics or HMDB51 or UCF101, you can run scripts/run_test.sh. For example:

# test
bash scripts/run_test.sh  ./configs/k400/k400_test.yaml

We provide several examples to do zero-shot validation on kinetics-400, UCF101 and HMDB51.

• To do zero-shot validation on Kinetics from CLIP pretrained models, you can run:

# zero-shot
bash scripts/run_test.sh  ./configs/k400/k400_ft_zero_shot.yaml

• To do zero-shot validation on UCF101 and HMDB51 from Kinetics pretrained models, you need first prepare the k400 pretrained model and then you can run:

# zero-shot
bash scripts/run_test.sh  ./configs/hmdb51/hmdb_ft_zero_shot.yaml

We provided several examples to train ActionCLIP with this repo:

• To train on Kinetics from CLIP pretrained models, you can run:

# train 
bash scripts/run_train.sh  ./configs/k400/k400_train.yaml

• To train on HMDB51 from Kinetics400 pretrained models, you can run:

# train 
bash scripts/run_train.sh  ./configs/hmdb51/hmdb_train.yaml

• To train on UCF101 from Kinetics400 pretrained models, you can run:

# train 
bash scripts/run_train.sh  ./configs/ucf101/ucf_train.yaml

More training details, you can find in configs/README.md

ActionCLIP is written and maintained by Mengmeng Wang and Jiazheng Xing.

If you find ActionClip useful in your research, please cite our paper.

Our code is based on CLIP and STM.