This is the official PyTorch implementation of QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models.

By Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang.

We propose QLLM, an accurate and efficient low-bitwidth post-training quantization method designed for LLMs.

• [10-03-2024] Release the code!🌟
• [17-01-2024] QLLM is accepted by ICLR 2024! 👏

• Install
• Usage
• Results
• Citation
• License
• Acknowledgement

conda create -n qllm python=3.10 -y
conda activate qllm
git clone https://github.com/ModelTC/QLLM
cd QLLM
pip install --upgrade pip 
pip install -e .

We provide the training scripts in scripts folder. For example, to perform W4A8 quantization for LLaMA-7B, run

sh scripts/llama-7b/w4a4.sh

Remember to change the path of model model and output path output_dir.

• QLLM achieve SoTA performance in weight-activation quantization

If you find our QLLM useful in your research, please consider to cite the following related papers:

@inproceedings{liu2024qllm,
  title = {{QLLM}: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models},
  author = {Liu, Jing and Gong, Ruihao and Wei, Xiuying and Dong, Zhiwei and Cai, Jianfei and Zhuang, Bohan},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year = {2024},
}

This repository is released under the Apache 2.0 license as found in the LICENSE file.

This repository is built upon OmniQuant. We thank the authors for their open-sourced code.