Smarter LLM Post-Training Quantization using End Loss Guidance, boosting the performance of
state-of-the-art weight-only scalar, weight-only vector, and weight-and-activation quantization methods.

• May, 2025: GuidedQuant is accepted to ICML 2025.

GuidedQuant enhances LLM quantization by integrating gradient information from the end loss into the quantization objective, boosting the performance of SOTA weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce LNQ, a non-uniform scalar quantization algorithm which is guaranteed to monotonically decrease the quantization objective value.

1. Install the requirements (we used Python 3.11 / CUDA 12.4 / pip 25.1 version).

   pip install -r requirements.txt

2. Install the Any-Precision-LLM CUDA kernels.

   Install either from source (this might take a while),

   cd inference/ap_gemv
   bash install.sh

   or from pre-built binaries,

   # CUDA 12.4
   pip install ap-gemv -i https://jusjinuk.me/whl/cu124

You could easily load and test them using AnyPrecisionForCausalLM class, as shown in the following example (runs on one RTX 3090).

from any_precision.modules.AnyPrecisionForCausalLM import AnyPrecisionForCausalLM
from transformers import AutoTokenizer, TextStreamer
import torch
quantized_model_name = "jusjinuk/Llama-3.3-70B-Instruct-2bit-GuidedQuant-LNQ"
# Use float16 for Llama models, and bfloat16 for Qwen / Gemma models
dtype = torch.float16 if "llama" in quantized_model_name.lower() else torch.bfloat16
model = AnyPrecisionForCausalLM.from_quantized(quantized_model_name, torch_dtype=dtype)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_name)
streamer = TextStreamer(tokenizer)
prompt = "Write me a short and concise story about Harry, Ron, and Hermione.\n"
chat = [
    {"role": "system", "content": "You are a helpful assistant.\n"},
    {"role": "user", "content": prompt},
]
inputs = tokenizer.apply_chat_template(
    chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
).to(model.device)
model.generate(inputs, 
    max_new_tokens=200, do_sample=False, temperature=1.0, streamer=streamer, pad_token_id=tokenizer.eos_token_id
)

We provide a simple inference script (~80 LOC) that uses torch.compile with Hugging Face generate function, showing the speed-up of LNQ + GuidedQuant quantized model, using Any-Precision-LLM kernel (ap-gemv kernel). This example is inspired by the demo code of Any-Precision-LLM.

# pre-trained Llama-3.1-8B-Instruct
# ~43 tok/s in one RTX 3090
python inference_example.py
# LNQ + GuidedQuant quantized Llama-3.1-8B-Instruct (bits=2)
# ~130 tok/s in one RTX 3090
python inference_example.py -q

In the paper, we report the further-optimized throughput of each model obtained by fusing the Q/K/V layer and the Up/Gate layer within every Transformer block.

We provide the tokenized calibration data for Llama-2 and Llama-3 to reproduce the results in the paper.

bash scripts/download_calibration.sh

Below command saves the weight gradients and activation gradients (averaged into $NUM_GROUPS groups) and quantizes model with SqueezeLLM.

bash scripts/run_sqllm.sh $MODEL_NAME $BITS $NUM_GROUPS
# e.g., bash scripts/run_sqllm.sh meta-llama/Llama-2-7b-hf 2 4

Afterwards, for LNQ + GuidedQuant, run the following command.

bash scripts/run_lnq.sh $MODEL_NAME $BITS $NUM_GROUPS
# e.g., bash scripts/run_lnq.sh meta-llama/Llama-2-7b-hf 2 4

We currently support the following models: Qwen3 (dense), Gemma3, Llama 3, and Llama 2. To add support for other models, you will need to modify (but are not limited to) the following directories:

• any_precision/analyzer/architectures/
• any_precision/analyzer/splitted_models/

git clone https://github.com/Dao-AILab/fast-hadamard-transform 
cd fast-hadamard-transform
pip install .
cd ..
cd qtip/qtip-kernels
pip install .
cd ../..

pip install transformers==4.45.2

cd qtip
# Example: bash exps/lufree_noft_qtip.sh 1mad 7b 2
bash exps/lufree_noft_qtip.sh $METHOD $MODEL_SIZE $BITS

cd spin_quant
# Example: bash scripts/2_eval_ptq_guided_save_wikitext2_7b_g1.sh meta-llama/Llama-2-7b-hf 4 4 4 "rotation/7B_W16A4KV4_lr_1.5_seed_0/R.bin"
bash scripts/2_eval_ptq_guided_save_wikitext2_7b_g1.sh $MODEL_NAME $W_BITS $A_BITS $KV_BITS $ROTATION_PATH

Run the following command to evaluate the perplexity of the pre-trained / quantized / pre-quantized models.

python run_eval.py

Add --downstream option to evaluate on downstream tasks using lm-eval-harness library.

YAQA is a more recent quantization method that approximates the Fisher information matrix using Kronecker-decomposed matrices. Compared to GuidedQuant, YAQA achieves better performance but requires more GPU resources to quantize a model (e.g., approximately 6 hours on 8×H100 GPUs for the Llama-2-7B model).

Evaluation results based on the experimental setup described in their paper are summarized in this table.

This code is heavily based on the following repositories:

• Any-Precision-LLM
• QTIP
• SpinQuant
• AQLM
• Fast Hadamard Transform
• gpt-fast

We thank the authors for their open-source implementations and contributions to the community.

Please cite our paper if you find our work useful:

@inproceedings{kim2025guidedquant,
      title={GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance}, 
      author={Jinuk Kim and Marwa El Halabi and Wonpyo Park and Clemens JS Schaefer and Deokjae Lee and Yeonhong Park and Jae W. Lee and Hyun Oh Song},
      booktitle = {International Conference on Machine Learning (ICML)},
      year={2025},
}