UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning

Sule Bai^1,2, Mingxing Li², Yong Liu¹, Jing Tang², Haoji Zhang¹, Lei Sun², Xiangxiang Chu², Yansong Tang¹

¹Tsinghua University ²ALibaba Group

1. We propose UniVG-R1, a reasoning guided MLLM for universal visual grounding, which employs GRPO with a cold-start initialization to effectively enhance reasoning capabilities across multimodal contexts.
2. A high-quality CoT dataset is introduced, encompassing diverse tasks, each meticulously annotated with detailed reasoning chains to facilitate advanced reasoning-based grounding.
3. We identify a difficulty bias in GRPO training, and propose a difficulty-aware weight adjustment strategy. Experiments validate that GRPO equipped with this strategy consistently enhance the model performance.
4. Extensive experiments demonstrate that our model achieves state-of-the-art performance across multiple grounding benchmarks, showcasing its versatility and generalizability.

We provide an online demo on 🤗Huggingface for convenience.

If you want to test locally, you can use the demo.py.

Our code is based on VLM-R1, please follow their instrcutions to set up the environment.

conda create -n univg-r1 python=3.10
conda activate univg-r1
bash setup.sh

1. Download the Stage 1 CoT-SFT Data and the MGrounding-630k dataset.
2. We use the LLaMA-Factory for supervised fine-tuning. Please follow their instructions to set up the environment.

llamafactory-cli train sft/stage1_cotsft.yaml

1. Download the Stage 2 RL Data and the MGrounding-630k dataset.

bash src/open-r1-multimodal/run_scripts/run_grpo_univg.sh

• For evaluation on the MIG-Bench, please follow their instructions to set up the environment. And we provide the UniVG-R1 scripts in the eval folder. Please replace them in the Migician codebase.
• For zero-shot evaluation, please first download original data, including LISA, LLMSeg, ReVOS, ReasonVOS. Then download our extracted bounding box annotations here.

Our work is primarily based on Migician, VLM-R1, Visual-RFT, LLaMA-Factory, lmms-eval. We are sincerely grateful for their excellent works.

If you find our paper and code helpful for your research, please consider starring our repository ⭐ and citing our work ✏️.

@article{bai2025univg,
  title={UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning},
  author={Bai, Sule and Li, Mingxing and Liu, Yong and Tang, Jing and Zhang, Haoji and Sun, Lei and Chu, Xiangxiang and Tang, Yansong},
  journal={arXiv preprint arXiv:2505.14231},
  year={2025}
}