[Feature] Are there plans to support AWQ and torch compile?

[sitabulaixizawaluduo]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Are there plans to support AWQ and torch compile?

Related resources

No response

[merrymercy]

Yes. I think we should be able to do this with torchao.
cc @jerryzh168 @msaroufim

[jerryzh168]

yes, we have awq (https://github.com/pytorch/ao/tree/main/torchao/prototype/awq) and GPTQ (https://github.com/pytorch/ao/blob/main/torchao/quantization/GPTQ_MT.py) implementations, both are compatible with torch.compile I think

code example for awq (uintx but only int4wo has speedup I think): https://github.com/pytorch/ao/blob/06ad55acb0d034a4e98e82a9eeddbd41d4d94b31/torchao/_models/llama/generate.py#L258-L284

code example for gptq (int4wo): https://github.com/pytorch/ao/blob/06ad55acb0d034a4e98e82a9eeddbd41d4d94b31/torchao/_models/llama/eval.py#L105-L127

[jerryzh168]

this requires full model quantization though I think, so we have change our current way of integrating torchao (which is applied per layer)