Add latest AWQ CUDA fp16 int4 kernels #198
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This new packed Tensor applies the formatting expected by the AWQ GEMM/GEMV kernels of first and second generation.
Before this change the weight Tensor was quantized just to be able to identify its target class before deserializing its inner tensors and meta. The target class is now explicitly identified in QModuleMixin.
This adds AWQBitsTensor, a QBitsTensor subclass that stores its inner Tensors as expected by AWQ CUDA kernels. A new `create` static method is added to QBitsTensor to select the best subclass when creating a QBitsTensor. This new method is now called automatically when quantizing or copying a QBitsTensor to a device. Note that it is still possible to create a QBitsTensor base class on any device by calling explicitly its constructor instead of the `create` method. This is in particular useful when reloading a serialized QBitsTensor to a specific device. In that case, another helper is added to optimize the QBitsTensor after deserialization.
These kernels are from: https://github.com/mit-han-lab/llm-awq but actually first implemented in: https://github.com/NVIDIA/TensorRT-LLM.
transformers version 4.41.0 refuses to load the custom MLP MNIST model.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This adds the latest AWQ CUDA kernels from https://github.com/mit-han-lab/llm-awq to the library.
@inproceedings{lin2023awq,
title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
booktitle={MLSys},
year={2024}
}
This also completely refactors the
QBitsTensorclass to allow it to be subclassed with tensors that store their underlying data, scale and zeropoint as expected by custom kernels.The latency of models quantized with
qint4weights is drastically reduced (almost divided by two). An upcoming pull-request will update the benchmark section with new numbers.Implements #111