[Feature]: qwen2.5 omni doesn't support bnb quantification.

[dylanliu2]

🚀 The feature, motivation and pitch

I'm trying to deploy qwen2.5-omni-3B on my laptop but found that it doesn't support bnb quantization. It seems that there is no other quantization version of qwen2.5-omni-3B that vllm can support now.
Will bnb quantization for qwen2.5-omni be supported in the furture?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

I will check this ASAP

[jeejeelee]

Are you using BNB online quantization?

[dylanliu2]

Are you using BNB online quantization?

I tried online quantization and also loading a bnb-quantized model. Both didn't work. It seems that "packed_modules_mapping" is missing.

[jeejeelee]

Online BNB quantization could indeed be supported, but offline BNB models might not be supportable.

[dylanliu2]

Online BNB quantization could indeed be supported, but offline BNB models might not be supportable.

Ok, good. So I just need to wait for the next version of vllm, right?

[dylanliu2]

Is there someone planning to fix this?

[jeejeelee]

Sorry for the delay feedback, I will implment this ASAP.