This repository contains the implementation of our paper Bootstrapping Language Models via DPO Implicit Rewards. We show that the implicit reward model from the prior DPO training can be utilized to bootstrap and further align LLMs.

• Bootstrapping with DPO Implicit Rewards (DICE)
  • Quick links
  • Base Models and Released Models
  • Setup
    • Install dependencies
    • Setup the bash script
  • Training scripts
  • Acknowledgement
  • Citation

Please refer to pipeline.sh#1.1_response_generation on instructions for batch inference with the appropriate chat template.

Please install dependencies using the following command:

git clone https://github.com/sail-sg/dice.git
conda create -n dice python=3.10
conda activate dice
cd dice/llama-factory
pip install -e .[deepspeed,metrics,bitsandbytes]
cd ..
pip install -e .
pip install -r requirements.txt
# optional to install flash attention
pip install flash-attn --no-build-isolation

Provide the local path to this repo to DICE_DIR in two files:

• scripts/run_dice/iter.sh
• scripts/run_dice/pipeline.sh

E.g. DICE_DIR="/home/username/dice"

We provide sample training scripts for both Llama3 and Zephyr settings. It is recommended to run the script with 8x A100 GPUs. For other hardware environments, you might need to adjust the script.

• Llama3

  bash scripts/run_dice/iter.sh llama3

• Zephyr

  bash scripts/run_dice/iter.sh zephyr

This repo is built on LLaMA-Factory. Thanks for the amazing work!

Please consider citing our paper if you find the repo helpful in your work:

@inproceedings{chen2025bootstrapping,
   title={Bootstrapping Language Models with DPO Implicit Rewards},
   author={Chen, Changyu and Liu, Zichen and Du, Chao and Pang, Tianyu and Liu, Qian and Sinha, Arunesh and Varakantham, Pradeep and Lin, Min},
   booktitle={International Conference on Learning Representations (ICLR)},
   year={2025}
}