Yicheng Chen, Xiangtai Li, Yining Li, Yanhong Zeng, Jianzong Wu, Xiangyu Zhao, Kai Chen

• [2024/06] Our paper Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language is released.

Auto Cherry Picker is a innovative framework designed to synthesize training samples for both perception and multi-modal reasoning tasks from a simple object list in natural language. It employs a nowly designed metric, CLIS, to ensure the quality of the synthetic data.

Python 3.10

Pytorch 2.3.0

pip install -r requirements.txt

Download Qwen1.5-14B-Chat

git clone https://huggingface.co/Qwen/Qwen1.5-14B-Chat

You can try other LLMs as Scene Graph Generator, and add it in the config/model_config.json.

• Step 1: Download InstanceDiffusion

git clone https://github.com/frank-xwang/InstanceDiffusion.git

• Step 2: Download model weights

Please download the pretrained InstanceDiffusion from Hugging Face or Google Drive and SD1.5, place them under InstanceDiffusion/pretrained folder.

Then, create a soft link under ACP folder.

ln -s InstanceDiffusion/pretrained ./pretrained

• Step 3: Download CLIP

git clone https://huggingface.co/openai/clip-vit-large-patch14

• Step 4: Download SDXL Refiner (Optional)

git clone https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0

To disable SDXL, you can set args.cascade_strength at infer_image.py to 0.

Please download Qwen-VL-Chat

git clone https://huggingface.co/Qwen/Qwen-VL-Chat

Please construct example pool for CLIS-L.

Download sim_map.json and relations_one_to_one.json under config/eval/

Download SAM model weights at Github

mkdir sam
cd sam
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

python inference_single_data.py

You can custom object list at inputs/demo.json. The generated images are under images/ and the synthesis training sample is under syn_data/.