Ye Liu^1†, Kevin Qinghong Lin^2†, Chang Wen Chen¹, Mike Zheng Shou²

¹The Hong Kong Polytechnic University ²Show Lab, National University of Singapore

TL;DR: Pioneer DeepSearch-like Video Understanding.

VideoMind is a multi-modal agent framework that enhances video reasoning by emulating human-like processes, such as breaking down tasks, localizing and verifying moments, and synthesizing answers. This approach addresses the unique challenges of temporal-grounded reasoning in a progressive strategy.

• 2025.04.05 📊 See BENCHMARK.md for evaluation results of VideoMind on public benchmarks.
• 2025.03.28 🚀 VideoMind-2B is ready on Hugging Face Spaces. Check it out!
• 2025.03.21 ⭐️ Code, model, and dataset release.
• 2025.03.17 🎉 Our tech report is available online.

ZS and FT refer to zero-shot and fine-tuned settings, respectively. * means third-party results.

See BENCHMARK.md for full evaluation results.

Play with our online demo or see DEMO.md for guidelines about how to deploy it locally.

We provide raw videos, compressed videos, and pre-processed annotations of 27 video grounding / QA datasets, including our VideoMind-SFT (481K) for training and multiple benchmarks for evaluation. We also release the datasets used during our early exploration (but not included in the final version) to facilitate future research.

The list of source datasets is shown below. See our dataset repo for more details.

The following datasets are not used in our project (partially used during early exploration), but we still share them to facilitate future research.

Notes:

1. For some datasets (e.g., ReXTime), the annotations and videos are stored in different folders. All the directories in Processed need to be downloaded.
2. Use the following commands to concatenate and extract video tar splits (e.g., videos.tar.gz.00, videos_3fps_480_noaudio.tar.gz.00).

# videos.tar.gz.00, videos.tar.gz.01
cat videos.tar.gz.* | tar -zxvf -
# videos_3fps_480_noaudio.tar.gz.00, videos_3fps_480_noaudio.tar.gz.01
cat videos_3fps_480_noaudio.tar.gz.* | tar -zxvf -

Our codebase supports training and evaluating on 27 video datasets and benchmarks with the following features.

• Flexible hardware settings: NVIDIA GPU / Ascend NPU, Single-Node / Multi-Node
• Efficient training techniques: DeepSpeed ZeRO, BF16, LoRA, SDPA, FlashAttention2, Liger-Kernel
• Customizing the base LLM and conversation templates
• Monitoring the training process via Tensorboard / Wandb
• Group sampling for mixed dataset training
• Multi-process / multi-device evaluation on public benchmarks

See TRAIN.md for a quick start guide.

See EVAL.md for details about evaluating VideoMind on public benchmarks.

Please kindly cite our paper if you find this project helpful.

@article{liu2025videomind,
  title={VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning},
  author={Liu, Ye and Lin, Kevin Qinghong and Chen, Chang Wen and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2503.13444},
  year={2025}
}