Error when loading GPTQ-quantized Mistral-Small model with vLLM

[VAmblardPEReN]

Hi,

Just tried to quantize Mistral-Small-3.1-24B with GPTQ using your example script.

Everything went smoothly but when I tried to load the model with vLLM, I got the following error message

ValueError: There is no module or parameter named 'multi_modal_projector.patch_merger.merging_layer.weight_packed' in Mistral3ForConditionalGeneration

I'm using one of the latest commits on the main branch of vLLM (0.9.2rc2.dev269+gf29fd8a7f) and same for llm-compressor (llmcompressor-0.6.1.dev35+g53240c63).

Thanks!

[dsikka]

@VAmblardPEReN do you mind sharing your model through hf hub?

[kylesayrs]

This is likely related to vllm-project/vllm#20046 where vllm doesn't properly map the ignore list unless SupportsQuant is added to Mistral3ForConditionalGeneration.

I believe if the SupportsQuant mixin is added to Mistral3ForConditionalGeneration, this issue will be resolved.

[VAmblardPEReN]

@VAmblardPEReN do you mind sharing your model through hf hub?

Sure, here you are: https://huggingface.co/VAmblardPEReN/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GPTQ!

[CAlexander0614]

I am having a similar issue loading the quantized model for serving:

ValueError: There is no module or parameter named 'multi_modal_projector.patch_merger.merging_layer.weight_scale' in Mistral3ForConditionalGeneration

Is there any fix or update on this?

I am running the latest nightly vllm openai docker container

[brian-dellabetta]

ValueError: There is no module or parameter named 'multi_modal_projector.patch_merger.merging_layer.weight_scale' in Mistral3ForConditionalGeneration

Isn't this error because vllm does not support quantized vision_tower / multi_modal_projector components? Even though those are excluded in the example script, the checkpoint at https://huggingface.co/VAmblardPEReN/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GPTQ shows quantized layers inside vision_tower / multi_modal_projector.

I am not sure why they aren't being excluded, but I think that's the reason for the error.

[CAlexander0614]

ValueError: There is no module or parameter named 'multi_modal_projector.patch_merger.merging_layer.weight_scale' in Mistral3ForConditionalGeneration

Isn't this error because vllm does not support quantized vision_tower / multi_modal_projector components? Even though those are excluded in the example script, the checkpoint at https://huggingface.co/VAmblardPEReN/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GPTQ shows quantized layers inside vision_tower / multi_modal_projector.

I am not sure why they aren't being excluded, but I think that's the reason for the error.

Interesting. I mean it would make sense...

[brian-dellabetta]

I ran mistral3_example.py and I do see that it is targeting layers we want to ignore:

2025-09-26T20:02:43.571160+0000 | compress_modules | INFO - Quantizing model.vision_tower.transformer.layers.4.feed_forward.gate_proj using 512 samples

This is because we are ignoring re:vision_tower.* and not re:.*vision_tower.*, so model.vision_tower isn't being caught.

I will open a quick PR to fix this

[CAlexander0614]

Thank you for your help!