VisVM

Official codebase for paper "Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension".

Prepare environment

conda env create -f environment.yml
cp ./utils/modeling_llava_next.py ~/.conda/envs/visvm/lib/python3.10/site-packages/transformers/models/llava_next/
cp ./utils/trainer/td_trainer.py ~/.conda/envs/visvm/lib/python3.10/site-packages/trl/trainer/
cp ./utils/__init__.py ~/.conda/envs/visvm/lib/python3.10/site-packages/trl/
cp ./utils/trainer/__init__.py ~/.conda/envs/visvm/lib/python3.10/site-packages/trl/trainer/

Generate responses:

bash ./script/batch_generate.sh

Prepare TD training data:

bash ./script/clip_score.sh

Train value model:

bash ./script/train_value.sh

SFT VLM:

bash ./script/train_sft.sh

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
script		script
utils		utils
README.md		README.md
batch_generate.py		batch_generate.py
control_decoding.py		control_decoding.py
environment.yml		environment.yml
generate_clip_score.py		generate_clip_score.py
sft_training.py		sft_training.py
value_training.py		value_training.py
vlm_value_models.py		vlm_value_models.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

VisVM

Prepare environment

Generate responses:

Prepare TD training data:

Train value model:

SFT VLM:

About

Uh oh!

Releases

Packages

Languages

si0wang/VisVM

Folders and files

Latest commit

History

Repository files navigation

VisVM

Prepare environment

Generate responses:

Prepare TD training data:

Train value model:

SFT VLM:

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages