Add AWQ (Activation-aware Weight Quantization) for llama, llama2, mpt, and mistral models

[namtranase]

AWQ: Activation-aware Weight Quantization for LLM - version apply to llamacpp

[Paper][Original Repo][Easy-to-use Repo]

Supported models:

• LLaMA
• LLaMA 2
• MPT
• Mistral v0.1

For more information, please refer to README.md

[BadisG]

Do you think it could work for MoE models like Mixtral aswell? I saw your perplexity table and it looks promising, it really improves the regular GGUF quants.

[namtranase]

Yes @BadisG I think it can work for Mixtral as well, and I'm working on it

[ggerganov]

This function can be merged into the existing llm_build_ffn

The ggml_div operator can optionally be applied if act_scales is non-NULL

[namtranase]

Thank you for your feedback, for more detail, only MPT models need an additional scale for gelu activation (original code). I will merge llm_build_ffn_mpt_awq to the llm_build_ffn but need to add the act_scales param (only for MPT and NULL for other models)
Llama, Llama2, and Mistral models do not need the scaling activation so they don't need any modification.

[ggerganov]

Don't think this has to be an hparam. We can determine whether to apply ffn_act or not based on it's presence in the model data

[namtranase]

I agree with you, but do you think there would be problems with the future modification of other models if they have scale layers like MPT

[ggerganov]

Unless I am missing some detail, it should work and can be extended to other models in a similar way. If ffn_act is NULL then no scaling is applied so we fallback to the original non-AWQ behaviour

[ggerganov]

I think an AWQ model should use the activation scales by default. AFAIU using an AWQ model with applying the activation scales would produce incorrect results because the FFN data has been scaled, so I don't see a reason to be able to turn off AWQ

[namtranase]

I just made an update based on your review @ggerganov

• remove use_awq arg
• check NULL for the ffn_act
• remove LLM_FFN_GELU_ACT
• merge llm_build_ffn_mpt_awq into llm_build_ffn

[ggerganov]

Thanks for the contribution!

Why does MPT requires the extra activation scales and the other models do not?

[casper-hansen]

@namtranase wow, this is great work! Did not think AWQ was suitable to get merged into llama.cpp, but here we are.

When adding model support, it is not uncommon that we modify the scaling functions to make new architectures work.

To maintain this compatibility, I would suggest apply_awq.py is replaced with imports from AutoAWQ as we are updating the framework regularly with help from Huggingface engineers as well. The only necessary function to keep looks like add_scale_weights.

Recent additions include Mixtral and LLaVa. Also working on Striped Hyena (which modifies the things in apply_awq.py).

CC: @ggerganov any thoughts on importing from AutoAWQ versus making PRs to modify code in apply_awq.py?

[casper-hansen]

Another question. It looks like the new awq-py is not compatible with any of @TheBloke's models because it expects an argument --awq-path which is not present in AutoAWQ. Wouldn't it make sense to update this awq-py module to be compatible with the 1000+ models already on HF?

For reference, one of the improvements I made from llm-awq to AutoAWQ is that you quantize everything in one go instead of the 2-step process. This eliminated the need for saving any intermediate files (e.g. the cache/search results).

[cebtenzzre]

Is there an explanation for why this PR removed mmap=True from convert-hf-to-gguf.py?

[namtranase]

@ggerganov I think because the MPT model needs to mul scale in the up_proj and later div scale in the act layer to cancel each other, while other models don't need it.

[namtranase]

Hi @casper-hansen, thank you for your reply and feedback. I already tried to reuse AutoAWQ but AFAIU, AutoAWQ only saves the quantized weights but not the scales, which is required by AWQ. I will try to make it compatible with AutoAWQ, which probably requires a PR, and will be adding more details when it is ready

[casper-hansen]

Hi @casper-hansen, thank you for your reply and feedback. I already tried to reuse AutoAWQ but AFAIU, AutoAWQ only saves the quantized weights but not the scales, which is required by AWQ. I will try to make it compatible with AutoAWQ, which probably requires a PR, and will be adding more details when it is ready

The weights you get from AutoAWQ are the quantized weights. You don’t need the scales with AutoAWQ as they are merged into the weights - that’s part of an improvement.

[JianbangZ]

Hi @casper-hansen, thank you for your reply and feedback. I already tried to reuse AutoAWQ but AFAIU, AutoAWQ only saves the quantized weights but not the scales, which is required by AWQ. I will try to make it compatible with AutoAWQ, which probably requires a PR, and will be adding more details when it is ready

The weights you get from AutoAWQ are the quantized weights. You don’t need the scales with AutoAWQ as they are merged into the weights - that’s part of an improvement.

I saw AutoAWQ only supports 4 bit. For MOE model, it's much more interesting to try out some lower bit quantization like gguf's Q2/Q3. Any plans to incorporate lower bit quantization in your AutoAWQ? what's the major challenges of enabling it?

[casper-hansen]

AutoAWQ lacks Triton/CUDA kernels for inference of other bit rates. The quantization is not really a problem, that’s just a matter of allowing it. This leaves me with the idea that maybe we could allow 3-bit quantization if we can export to GGUF and run inference with llama.cpp.

[JianbangZ]

AutoAWQ lacks Triton/CUDA kernels for inference of other bit rates. The quantization is not really a problem, that’s just a matter of allowing it. This leaves me with the idea that maybe we could allow 3-bit quantization if we can export to GGUF and run inference with llama.cpp.

As far as I understand, correct me if I am wrong

1. This PR grabs the statistical precomputed scales for different model architectures, and apply the scales to the weights. Then up to the gguf k-quant to handle the quantization using the awq scales embedded
2. AutoAWQ directly uses a calibration set (pile, wiki) to quantize to 4 bit.

Are you saying you want to directly convert a 3/4bit AWQ model quantized by AutoAWQ to gguf format? For a company project (without too much details for good reasons), we quantized a 7B model using GPTQ, and then convert GPTQ model to gguf q4_0 for less quantization loss. So should be the same thing.

[casper-hansen]

AutoAWQ lacks Triton/CUDA kernels for inference of other bit rates. The quantization is not really a problem, that’s just a matter of allowing it. This leaves me with the idea that maybe we could allow 3-bit quantization if we can export to GGUF and run inference with llama.cpp.

As far as I understand, correct me if I am wrong

1. This PR grabs the statistical precomputed scales for different model architectures, and apply the scales to the weights. Then up to the gguf k-quant to handle the quantization using the awq scales embedded
2. AutoAWQ directly uses a calibration set (pile, wiki) to quantize to 4 bit.

Are you saying you want to directly convert a 3/4bit AWQ model quantized by AutoAWQ to gguf format? For a company project (without too much details for good reasons), we quantized a 7B model using GPTQ, and then convert GPTQ model to gguf q4_0 for less quantization loss. So should be the same thing.

1. The original llm-awq was a 2-step process. Firstly, you calculate the scales and save them. Secondly, you load the scales and apply them along with the quantization method and packing to get the quantized model. AutoAWQ just skips saving the scales and goes directly to quantizing the model as saving the scales is unnecessary. The result is the same, but the latter is easier to work with. So GGUF should not handle any scales as they are completely redundant, it should just load the 4-bit AWQ model and convert it to GGUF.
2. Both llm-awq and AutoAWQ use a calibration set to create the scales. They are precisely equivalent and will give you the exact same perplexity - AutoAWQ was just created to be easier to work with and to enable optimized inference more easily.

The problem with the converter is that it does not work with quantized models, e.g. if you were to quantize the model with llm-awq or AutoAWQ, it would not work for conversion. The converter needs an FP16 model which you get when you just apply the scales and weight clipping. However, all the 1000+ models on Huggingface are INT4 models, so we should have compatibility to convert those models.