-
Notifications
You must be signed in to change notification settings - Fork 617
[quantization] Add w8a16 quantization support #4541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces support for W8A16 quantization for dense models. The changes include a new quantization method AscendW8A16LinearMethod and its integration into the existing quantization framework. The implementation is straightforward and looks correct. I have one suggestion to refactor the apply method in vllm_ascend/quantization/w8a16.py to reduce code duplication, which will improve readability and maintainability.
vllm_ascend/quantization/w8a16.py
Outdated
| if is_310p(): | ||
| # On 300I Duo platform, we need transpose again if | ||
| # using nz. This transpose can be skipped in torchair. | ||
| output = torch_npu.npu_weight_quant_batchmatmul( | ||
| x=x, | ||
| weight=layer.weight.data.transpose(0, 1), | ||
| antiquant_scale=layer.weight_scale, | ||
| antiquant_offset=layer.weight_offset, | ||
| bias=bias | ||
| ) | ||
| else: | ||
| output = torch_npu.npu_weight_quant_batchmatmul( | ||
| x=x, | ||
| weight=layer.weight, | ||
| antiquant_scale=layer.weight_scale, | ||
| antiquant_offset=layer.weight_offset, | ||
| bias=bias | ||
| ) | ||
| return output |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The if/else block contains duplicated calls to torch_npu.npu_weight_quant_batchmatmul. This can be refactored to improve readability and maintainability by determining the weight tensor first and then making a single call to the function.
weight = layer.weight
if is_310p():
# On 300I Duo platform, we need transpose again if
# using nz. This transpose can be skipped in torchair.
weight = layer.weight.data.transpose(0, 1)
output = torch_npu.npu_weight_quant_batchmatmul(
x=x,
weight=weight,
antiquant_scale=layer.weight_scale,
antiquant_offset=layer.weight_offset,
bias=bias
)
return output|
please fix DCO and lint, refer to https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html |
Signed-off-by: yyt <yangyit139@gmail.com>
Signed-off-by: yyt <yangyit139@gmail.com>
Signed-off-by: yyt <yangyit139@gmail.com>
8b8a945 to
c3c2b2f
Compare
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
What this PR does / why we need it?
related to #4267
Does this PR introduce any user-facing change?
support w8a16 quantization now
How was this patch tested?