intel
diff --git a/‎docs/source/quantization_weight_only.md
Copy file name to clipboard
+34-29Lines changed: 34 additions & 29 deletions b/‎docs/source/quantization_weight_only.md
Copy file name to clipboard
+34-29Lines changed: 34 additions & 29 deletions
diff --git a/‎examples/.config/model_params_onnxrt.json
Copy file name to clipboard
+15-8Lines changed: 15 additions & 8 deletions b/‎examples/.config/model_params_onnxrt.json
Copy file name to clipboard
+15-8Lines changed: 15 additions & 8 deletions
diff --git a/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/README.md
Copy file name to clipboard
+17-7Lines changed: 17 additions & 7 deletions b/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/README.md
Copy file name to clipboard
+17-7Lines changed: 17 additions & 7 deletions
diff --git a/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/main.py
Copy file name to clipboard
+1-1Lines changed: 1 addition & 1 deletion b/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/main.py
Copy file name to clipboard
+1-1Lines changed: 1 addition & 1 deletion
diff --git a/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/requirements.txt
Copy file name to clipboard
+3-2Lines changed: 3 additions & 2 deletions b/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/requirements.txt
Copy file name to clipboard
+3-2Lines changed: 3 additions & 2 deletions
diff --git a/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/run_quant.sh
Copy file name to clipboard
+1-1Lines changed: 1 addition & 1 deletion b/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/ptq_static/run_quant.sh
Copy file name to clipboard
+1-1Lines changed: 1 addition & 1 deletion
diff --git a/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/weight_only/README.md
Copy file name to clipboard
+19-5Lines changed: 19 additions & 5 deletions b/‎examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/weight_only/README.md
Copy file name to clipboard
+19-5Lines changed: 19 additions & 5 deletions
@@ -9,6 +9,8 @@ Weight Only Quantization (WOQ)
 
 4. [Layer Wise Quantization](#layer-wise-quantization)
 
+5. [WOQ Algorithms Tuning](#woq-algorithms-tuning)
+
 
 ## Introduction
 
@@ -142,35 +144,6 @@ The saved_results folder contains two files: `best_model.pt` and `qconfig.json`,
 To seek the performance of weight-only quantized models, Please go to [Intel Extension for Transformers](https://github.com/intel/intel-extension-for-transformers/tree/main/examples/huggingface/pytorch/text-generation/quantization#1-performance) to quantize and deploy the model.
 
 
-### **WOQ algorithms tuning**
-
-To find the best algorithm, users can omit specifying a particular algorithm. In comparison to setting a specific algorithm, this tuning process will traverse through a set of pre-defined WOQ configurations and identify the optimal one with the best result. For details usage, please refer to the [tuning strategy](./tuning_strategies.md#Basic).
-
-> **Note:** Currently, this behavior is specific to the `ONNX Runtime` backend.
-
-**Pre-defined configurations**
-
-| WOQ configurations | setting |
-|:------------------:|:-------:|
-|RTN_G32ASYM| {"algorithm": "RTN", "group_size": 32, "scheme": "asym"}|
-|GPTQ_G32ASYM| {"algorithm": "GPTQ", "group_size": 32, "scheme": "asym"}|
-|GPTQ_G32ASYM_DISABLE_LAST_MATMUL| {"algorithm": "GPTQ", "group_size": 32, "scheme": "asym"} <br> & disable last MatMul|
-|GPTQ_G128ASYM| {"algorithm": "GPTQ", "group_size": 128, "scheme": "asym"}|
-|AWQ_G32ASYM| {"algorithm": "AWQ", "group_size": 32, "scheme": "asym"}|
-
-**User code example**
-
-```python
-conf = PostTrainingQuantConfig(
-    approach="weight_only",
-    quant_level="auto",  # quant_level supports "auto" or 1 for woq config tuning
-)
-q_model = quantization.fit(model, conf, eval_func=eval_func, calib_dataloader=dataloader)
-q_model.save("saved_results")
-```
-
-Refer to this [link](../../examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/weight_only) for an example of WOQ algorithms tuning on ONNX Llama models.
-
 ## Layer Wise Quantization
 
 Large language models (LLMs) have shown exceptional performance across various tasks, meanwhile, the substantial parameter size poses significant challenges for deployment. Layer-wise quantization(LWQ) can greatly reduce the memory footprint of LLMs, usually 80-90% reduction, which means that users can quantize LLMs even on single node using GPU or CPU.  We can quantize the model under memory-constrained devices, therefore making the huge-sized LLM quantization possible.
@@ -213,6 +186,38 @@ q_model.save(ouput_dir)
 q_model = load(ouput_dir, fp32_model, weight_only=True, layer_wise=True)
 ```
 
+
+## WOQ Algorithms Tuning
+
+To find the best algorithm, users can omit specifying a particular algorithm. In comparison to setting a specific algorithm, this tuning process will traverse through a set of pre-defined WOQ configurations and identify the optimal one with the best result. For details usage, please refer to the [tuning strategy](./tuning_strategies.md#Basic).
+
+> **Note:** Currently, this behavior is specific to the `ONNX Runtime` backend.
+
+**Pre-defined configurations**
+
+| WOQ configurations | Comments |
+|:------------------:|:-------:|
+|RTN_G32ASYM| {"algorithm": "RTN", "group_size": 32, "scheme": "asym"}|
+|GPTQ_G32ASYM| {"algorithm": "GPTQ", "group_size": 32, "scheme": "asym"}|
+|GPTQ_G32ASYM_DISABLE_LAST_MATMUL| {"algorithm": "GPTQ", "group_size": 32, "scheme": "asym"} <br> & disable last MatMul|
+|GPTQ_G128ASYM| {"algorithm": "GPTQ", "group_size": 128, "scheme": "asym"}|
+|AWQ_G32ASYM| {"algorithm": "AWQ", "group_size": 32, "scheme": "asym"}|
+
+### **User code example**
+
+```python
+conf = PostTrainingQuantConfig(
+    approach="weight_only",
+    quant_level="auto",  # quant_level supports "auto" or 1 for woq config tuning
+)
+q_model = quantization.fit(model, conf, eval_func=eval_func, calib_dataloader=dataloader)
+q_model.save("saved_results")
+```
+
+Refer to this [link](../../examples/onnxrt/nlp/huggingface_model/text_generation/llama/quantization/weight_only) for an example of WOQ algorithms tuning on ONNX Llama models.
+
+
+
 ## Reference
 
 [1]. Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." arXiv preprint arXiv:2211.10438 (2022).
 
@@ -756,31 +756,38 @@
       "main_script": "main.py",
       "batch_size": 1
     },
-    "llama-7b": {
+    "llama-2-7b": {
       "model_src_dir": "nlp/huggingface_model/text_generation/llama/quantization/ptq_static",
       "dataset_location": "",
-      "input_model": "/tf_dataset2/models/onnx/llama-7b",
+      "input_model": "/tf_dataset2/models/onnx/llama-2-7b",
       "main_script": "main.py",
       "batch_size": 1
     },
-    "llama-7b-rtn": {
+    "llama-2-7b-rtn": {
       "model_src_dir": "nlp/huggingface_model/text_generation/llama/quantization/weight_only",
       "dataset_location": "",
-      "input_model": "/tf_dataset2/models/onnx/llama-7b",
+      "input_model": "/tf_dataset2/models/onnx/llama-2-7b",
       "main_script": "main.py",
       "batch_size": 1
     },
-    "llama-7b-awq": {
+    "llama-2-7b-awq": {
       "model_src_dir": "nlp/huggingface_model/text_generation/llama/quantization/weight_only",
       "dataset_location": "",
-      "input_model": "/tf_dataset2/models/onnx/llama-7b",
+      "input_model": "/tf_dataset2/models/onnx/llama-2-7b",
       "main_script": "main.py",
       "batch_size": 1
     },
-    "llama-7b-gptq": {
+    "llama-2-7b-gptq": {
       "model_src_dir": "nlp/huggingface_model/text_generation/llama/quantization/weight_only",
       "dataset_location": "",
-      "input_model": "/tf_dataset2/models/onnx/llama-7b",
+      "input_model": "/tf_dataset2/models/onnx/llama-2-7b",
+      "main_script": "main.py",
+      "batch_size": 1
+    },
+    "llama-2-7b-woq_tune": {
+      "model_src_dir": "nlp/huggingface_model/text_generation/llama/quantization/weight_only",
+      "dataset_location": "",
+      "input_model": "/tf_dataset2/models/onnx/llama-2-7b",
       "main_script": "main.py",
       "batch_size": 1
     },
 
@@ -8,16 +8,26 @@ This example confirms llama's accuracy and speed based on [lambada](https://hugg
 ## 1. Environment
 ```shell
 pip install neural-compressor
-pip install -r requirements.txt
+SKIP_RUNTIME=True pip install -r requirements.txt
 ```
 > Note: Validated ONNX Runtime [Version](/docs/source/installation_guide.md#validated-software-environment).
 
 ## 2. Prepare Model
 
+Note that this README.md uses meta-llama/Llama-2-7b-hf as an example. There are other models available that can be used for quantization. The following table shows a few models' configurations:
+
+| Model | Num Hidden Layers| Num Attention Heads | Hidden Size |
+| --- | --- | --- | --- |
+| [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 32 | 32 | 4096 |
+| [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 32 | 32 | 4096 |
+| [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) | 40 | 40 | 5120 |
+| [meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 40 | 40 | 5120 |
+| [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf) | 80 | 64 | 8192 |
+| [meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 80 | 64 | 8192 |
+
+Export to ONNX model:
 ```bash
-python prepare_model.py  --input_model="decapoda-research/llama-7b-hf" --output_model="./llama_7b"
-# or
-python prepare_model.py  --input_model="decapoda-research/llama-13b-hf" --output_model="./llama_13b"
+optimum-cli export onnx --model meta-llama/Llama-2-7b-hf --task text-generation-with-past ./Llama-2-7b-hf
 ```
 
 # Run
@@ -30,7 +40,7 @@ bash run_quant.sh --input_model=/path/to/model \ # folder path of onnx model
                   --batch_size=batch_size # optional \
                   --dataset NeelNanda/pile-10k \
                   --alpha 0.6 \ # 0.6 for llama-7b, 0.8 for llama-13b
-                  --tokenizer=decapoda-research/llama-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
+                  --tokenizer=meta-llama/Llama-2-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
                   --quant_format="QOperator" # or QDQ, optional
 ```
 
@@ -42,7 +52,7 @@ Accuracy:
 bash run_benchmark.sh --input_model=path/to/model \ # folder path of onnx model
                       --batch_size=batch_size \ # optional 
                       --mode=accuracy \
-                      --tokenizer=decapoda-research/llama-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
+                      --tokenizer=meta-llama/Llama-2-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
                       --tasks=lambada_openai
 ```
 
@@ -51,6 +61,6 @@ Performance:
 numactl -m 0 -C 0-3 bash run_benchmark.sh --input_model=path/to/model \ # folder path of onnx model
                                           --mode=performance \
                                           --batch_size=batch_size # optional \
-                                          --tokenizer=decapoda-research/llama-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
+                                          --tokenizer=meta-llama/Llama-2-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
                                           --intra_op_num_threads=4
 ```
@@ -72,7 +72,7 @@
     '--tokenizer',
     type=str,
     help="pretrained model name or path of tokenizer files",
-    default="decapoda-research/llama-7b-hf"
+    default="meta-llama/Llama-2-7b-hf"
 )
 parser.add_argument(
     '--workspace',
 
@@ -1,5 +1,3 @@
-git+https://github.com/EleutherAI/lm-evaluation-harness.git@83dbfbf6070324f3e5872f63e49d49ff7ef4c9b3
-intel-extension-for-transformers
 torch
 transformers
 accelerate
@@ -9,3 +7,6 @@ onnxruntime-extensions; python_version < '3.11'
 datasets
 optimum
 evaluate
+peft
+git+https://github.com/EleutherAI/lm-evaluation-harness.git@cc9778fbe4fa1a709be2abed9deb6180fd40e7e2
+git+https://github.com/intel/intel-extension-for-transformers.git@abce937aa70ed2e30fbdf0aefd38cc3dd9d1d32a
@@ -62,7 +62,7 @@ function run_tuning {
     python main.py \
             --quant_format ${quant_format-QOperator} \
             --model_path ${input_model} \
-	    --tokenizer ${tokenizer-decapoda-research/llama-7b-hf} \
+	    --tokenizer ${tokenizer-meta-llama/Llama-2-7b-hf} \
             --output_model ${output_model} \
             --batch_size ${batch_size-1} \
             --smooth_quant_alpha ${alpha-0.6} \
 
@@ -8,36 +8,50 @@ This example confirms llama's weight only accuracy on [lambada](https://huggingf
 ## 1. Environment
 ```shell
 pip install neural-compressor
-pip install -r requirements.txt
+SKIP_RUNTIME=True pip install -r requirements.txt
 ```
 > Note: Validated ONNX Runtime [Version](/docs/source/installation_guide.md#validated-software-environment).
 
 > Note: Weight-only quantization in Intel® Neural Compressor is still under development. We encourage you to use the `master` branch to access the latest features.
 
 ## 2. Prepare Model
 
+Note that this README.md uses meta-llama/Llama-2-7b-hf as an example. There are other models available that can be used for weight-only quantization. The following table shows a few models' configurations:
+
+| Model | Num Hidden Layers| Num Attention Heads | Hidden Size |
+| --- | --- | --- | --- |
+| [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 32 | 32 | 4096 |
+| [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 32 | 32 | 4096 |
+| [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) | 40 | 40 | 5120 |
+| [meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 40 | 40 | 5120 |
+| [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf) | 80 | 64 | 8192 |
+| [meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 80 | 64 | 8192 |
+
+Export to ONNX model:
 ```bash
-python prepare_model.py  --input_model="decapoda-research/llama-7b-hf" --output_model="./llama_7b"
+python prepare_model.py  --input_model="meta-llama/Llama-2-7b-hf" --output_model="./llama-2-7b-hf"
 ```
 
 # Run
 
 ## 1. Quantization
 
+Set `algorithm=WOQ_TUNE` to tune weight-only quantization algorithm or specify algorithm to `RTN` or `GPTQ` or `AWQ`.
+
 ```bash
 bash run_quant.sh --input_model=/path/to/model \ # folder path of onnx model
                   --output_model=/path/to/model_tune \ # folder path to save onnx model
                   --batch_size=batch_size # optional \
                   --dataset=NeelNanda/pile-10k \
-                  --tokenizer=decapoda-research/llama-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
-                  --algorithm=RTN # support RTN, AWQ, GPTQ
+                  --tokenizer=meta-llama/Llama-2-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
+                  --algorithm=WOQ_TUNE # support WOQ_TUNE, RTN, AWQ, GPTQ
 ```
 
 ## 2. Benchmark
 
 ```bash
 bash run_benchmark.sh --input_model=path/to/model \ # folder path of onnx model
                       --batch_size=batch_size \ # optional 
-                      --tokenizer=decapoda-research/llama-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
+                      --tokenizer=meta-llama/Llama-2-7b-hf \ # model name or folder path containing all relevant files for model's tokenizer
                       --tasks=lambada_openai
 ```
Original file line number	Diff line number	Diff line change
`@@ -72,7 +72,7 @@`
`72`	`72`	`'--tokenizer',`
`73`	`73`	`type=str,`
`74`	`74`	`help="pretrained model name or path of tokenizer files",`
`75`		`- default="decapoda-research/llama-7b-hf"`
	`75`	`+ default="meta-llama/Llama-2-7b-hf"`
`76`	`76`	`)`
`77`	`77`	`parser.add_argument(`
`78`	`78`	`'--workspace',`