[Core]Add Ascend Quantize #7
Conversation
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
vllm_ascend/ops/layernorm.py
Outdated
|
|
||
| def forward_oot( | ||
| self, | ||
| x: torch.Tensor, | ||
| residual: Optional[torch.Tensor] = None, | ||
| ) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]: | ||
| if hasattr(self, "module"): | ||
| return self.module.forward_anti_outlier(x, residual) |
There was a problem hiding this comment.
does self.module only used here? If yes, how about something like:
try:
from mindie_turbo import RMSNormWithAntiOutlier
except:
RMSNormWithAntiOutlier = None
def forward_oot():
if RMSNormWithAntiOutlier is not None:
return RMSNormWithAntiOutlier(self.hidden_size).forward_anti_outlier(x, residual)
....
Not sure enable_rmsnorm_with_antioutlier is need, it seems only added a new self.module there.
There was a problem hiding this comment.
Details of RMSNormWithAntiOutlier are moved out of vllm_ascend. There's no need to change the implemantation of rmsnorm in vllm_ascend now.
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
|
|
||
| return MindIETurboQuantizer.get_quantizer(quant_config) | ||
| except Exception: | ||
| raise NotImplementedError("There is no available ascend quantizer.") |
There was a problem hiding this comment.
please use import_lib to check if mindie_turbo is available or not. The try cache here is too large.
| ] | ||
|
|
||
|
|
||
| @pytest.mark.skipif(not is_mindie_turbo_supported(), |
There was a problem hiding this comment.
Please add a TODO here. Once more method is available in vllm-ascend. the skip can be removed
There was a problem hiding this comment.
This test case is designed for quantization methods based on mindie-turbo. For other possible quantization methods in the future, we can add new test cases.
|
|
||
| import vllm_ascend # noqa: F401 | ||
| from vllm_ascend.quantization.quant_config import AscendLinearMethod | ||
|
|
There was a problem hiding this comment.
why import inner the test?
There was a problem hiding this comment.
This is because mindie_turbo should be import before vllm in early versions of mindie_turbo. Perhaps this conflict has been resolved now and these packages imported can be moved outside.
|
|
||
| # When not using anti-outlier algorithms, "anti_method" refers to an empty string. | ||
| if len(quant_config["anti_method"]) > 0: | ||
| enable_rmsnorm_with_antioutlier() |
There was a problem hiding this comment.
In my perspective, this looks kind of strange, this interface seems very detail and specific, Is it possible for you to hide more detail under the hood? I believe this part can be wrote in more general way.
There was a problem hiding this comment.
Details of RMSNormWithAntiOutlier are moved out of vllm_ascend. Related codes will be hidden into mindie_turbo.
…quantizer Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
Signed-off-by: angazenn <zengyanjia@huawei.com>
| from tests.quantization.utils import is_mindie_turbo_supported, example_quantization | ||
|
|
||
| MODELS = [ | ||
| "/home/zyj/data/Qwen2.5-0.5B-Instruct/", |
There was a problem hiding this comment.
this is an mistake. changed to Qwen/Qwen2.5-0.5B-Instruct now
vllm_ascend/platform.py
Outdated
| """ | ||
| Do some pre-registeration or update action for ascend platform. | ||
| """ | ||
| from vllm_ascend.quantization.quant_config import AscendQuantConfig # noqa: F401 |
There was a problem hiding this comment.
vllm-project/vllm#12432 is not merged. Maybe you can move this import to register function in __init__.py, but I'm not sure if it will lead the circle import error or not. You can have a try first.
There was a problem hiding this comment.
it seems that there are circular import if moving this import. But the codes still work and whole inference process can generate correct texts, so I move this codes to register function
Signed-off-by: angazenn <zengyanjia@huawei.com>
This pr adds ascend quantization interface to vllm-ascend, including AscendQuantConfig class which inherits from vllm QuantizationConfig class, AscendLinearMethod class which inherits from vllm LinearMethodBase class, AscendQuantizer class that dispatches corresponding quanzation methods.