[Fix] fix quantization arg when using marlin

[DreamTeamWangbowen]

#3331 fix when using marlin model

[DreamTeamWangbowen]

@zhuohan123 @WoosukKwon please, could you help me to merge it?

[DreamTeamWangbowen]

After the fix is merged, it can run normally.

[esmeetu]

Thanks for your contribution! @DreamTeamWangbowen Could you fix the code style following CONTRIBUTING.md ?

[DreamTeamWangbowen]

Thanks for your contribution! @DreamTeamWangbowen Could you fix the code style following CONTRIBUTING.md ?

Okay, I submit a new issue and fix it.

[esmeetu]

please use sh format.sh to format your code. And then i can merge this.

[WoosukKwon]

@DreamTeamWangbowen Do we need this btw? IIUC, the Marlin kernel is automatically used for GPTQ models when the condition is met (act_order=False, etc.).

[DreamTeamWangbowen]

please use sh format.sh to format your code. And then i can merge this.

I've finished formatting my code.

[DreamTeamWangbowen]

@DreamTeamWangbowen Do we need this btw? IIUC, the Marlin kernel is automatically used for GPTQ models when the condition is met (act_order=False, etc.).

Yes, we need it, I did not find the act_order parameter in the code and model configuration file.

The model address I use is https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

[WoosukKwon]

Yes, we need it, I did not find the act_order parameter in the code and model configuration file.
The model address I use is https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

@DreamTeamWangbowen IIUC, the Marlin kernel should be automatically used (without specifying quantization=marlin):

vllm/vllm/config.py

Lines 174 to 176 in 654865e

	if (hf_quant_method == "gptq"
	and "is_marlin_format" in hf_quant_config
	and hf_quant_config["is_marlin_format"]):

While the condition was not actually about act_order (sorry for the wrong information), the mode configuration file meets the above condition.

[DreamTeamWangbowen]

Yes, we need it, I did not find the act_order parameter in the code and model configuration file.
The model address I use is https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

@DreamTeamWangbowen IIUC, the Marlin kernel should be automatically used (without specifying quantization=marlin):

vllm/vllm/config.py

Lines 174 to 176 in 654865e

	if (hf_quant_method == "gptq"
	and "is_marlin_format" in hf_quant_config
	and hf_quant_config["is_marlin_format"]):

While the condition was not actually about act_order (sorry for the wrong information), the mode configuration file meets the above condition.

If I do not specify marlin, but specify gptq, the following error will occur

File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 130, in init
self._verify_quantization()
File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 204, in _verify_quantization
raise ValueError(
ValueError: Quantization method specified in the model config (marlin) does not match the quantization method specified in the quantization argument (gptq).

[DreamTeamWangbowen]

Yes, we need it, I did not find the act_order parameter in the code and model configuration file.
The model address I use is https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

@DreamTeamWangbowen IIUC, the Marlin kernel should be automatically used (without specifying quantization=marlin):

vllm/vllm/config.py

Lines 174 to 176 in 654865e

	if (hf_quant_method == "gptq"
	and "is_marlin_format" in hf_quant_config
	and hf_quant_config["is_marlin_format"]):

While the condition was not actually about act_order (sorry for the wrong information), the mode configuration file meets the above condition.

Look at the code here 178~185, there is a judgment here

vllm/vllm/config.py

Lines 178 to 185 in 654865e

	if self.quantization is None:
	self.quantization = hf_quant_method
	elif self.quantization != hf_quant_method:
	raise ValueError(
	"Quantization method specified in the model config "
	f"({hf_quant_method}) does not match the quantization "
	f"method specified in the `quantization` argument "
	f"({self.quantization}).")

Therefore we need to add marlin to the quantization argument.

There may be another way to modify it, or add and change self.quantization="gptq" to self.quantization="marlin" on line 177

vllm/vllm/config.py

Line 177 in 654865e

hf_quant_method = "marlin"

[esmeetu]

@DreamTeamWangbowen I think you can run your server without pass -q or --quantization args since vLLM will detect quant method from config.json.

[DreamTeamWangbowen]

@DreamTeamWangbowen I think you can run your server without pass -q or --quantization args since vLLM will detect quant method from config.json.

Yes, but if the quantization parameter is specified as gptq or marlin, an error will occur when the server is running.

[robertgshaw2-redhat]

@DreamTeamWangbowen

Marlin kernels use a special serialization method that are different from exllama. So the model must be saved on disk in Marlin format to be loaded by vLLM. vLLM currently does not support converting formats on the fly (though this is something we are working on). The model must be saved to disk in Marlin format to run in vLLM.

I added the functionality to save models in Marlin format to AutoGPTQ. Here's an example here:

• https://github.com/robertgshaw2-neuralmagic/marlin-example/blob/master/apply_gptq_save_marlin.py

Also here's a model I saved in this format:

• https://huggingface.co/neuralmagic/zephyr-7b-beta-marlin

I will add a doc to vLLM about this

Note --> Marlin currently requires group_size=128 act_order=False. We are working on expanding this

@WoosukKwon FYI

Passing marlin to this argument will not work

[DreamTeamWangbowen]

@DreamTeamWangbowen

Marlin kernels use a special serialization method that are different from exllama. So the model must be saved on disk in Marlin format to be loaded by vLLM. vLLM currently does not support converting formats on the fly (though this is something we are working on). The model must be saved to disk in Marlin format to run in vLLM.

I added the functionality to save models in Marlin format to AutoGPTQ. Here's an example here:

• https://github.com/robertgshaw2-neuralmagic/marlin-example/blob/master/apply_gptq_save_marlin.py

Also here's a model I saved in this format:

• https://huggingface.co/neuralmagic/zephyr-7b-beta-marlin

I will add a doc to vLLM about this

Note --> Marlin currently requires group_size=128 act_order=False. We are working on expanding this

@WoosukKwon FYI

Passing marlin to this argument will not work

yeath, the model I use is in marlin format https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

[robertgshaw2-redhat]

@DreamTeamWangbowen
Marlin kernels use a special serialization method that are different from exllama. So the model must be saved on disk in Marlin format to be loaded by vLLM. vLLM currently does not support converting formats on the fly (though this is something we are working on). The model must be saved to disk in Marlin format to run in vLLM.
I added the functionality to save models in Marlin format to AutoGPTQ. Here's an example here:

• https://github.com/robertgshaw2-neuralmagic/marlin-example/blob/master/apply_gptq_save_marlin.py

Also here's a model I saved in this format:

• https://huggingface.co/neuralmagic/zephyr-7b-beta-marlin

I will add a doc to vLLM about this
Note --> Marlin currently requires group_size=128 act_order=False. We are working on expanding this
@WoosukKwon FYI
Passing marlin to this argument will not work

yeath, the model I use is in marlin format https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

Nice - vllm will use marlin by default if you pass this model. You do not need to pass the --quantization argument explicitly

[DreamTeamWangbowen]

@DreamTeamWangbowen
Marlin kernels use a special serialization method that are different from exllama. So the model must be saved on disk in Marlin format to be loaded by vLLM. vLLM currently does not support converting formats on the fly (though this is something we are working on). The model must be saved to disk in Marlin format to run in vLLM.
I added the functionality to save models in Marlin format to AutoGPTQ. Here's an example here:

• https://github.com/robertgshaw2-neuralmagic/marlin-example/blob/master/apply_gptq_save_marlin.py

Also here's a model I saved in this format:

• https://huggingface.co/neuralmagic/zephyr-7b-beta-marlin

I will add a doc to vLLM about this
Note --> Marlin currently requires group_size=128 act_order=False. We are working on expanding this
@WoosukKwon FYI
Passing marlin to this argument will not work

yeath, the model I use is in marlin format https://huggingface.co/neuralmagic/Nous-Hermes-2-Yi-34B-marlin

Nice - vllm will use marlin by default if you pass this model. You do not need to pass the --quantization argument explicitly

Yes, the marlin model I am using is act_order=False and group_size=128.

But I understand that the -q argument does not tell vllm to specify the type of quantization method used, such as awq, gptq, so I added marlin, for example tell vllm to use MarlinLinearMethod

[WoosukKwon]

@DreamTeamWangbowen IIUC, Marlin is not a quantization method. It's a fast kernel implementation for GPTQ.

I've updated the PR so that it fixes the bug that quantization="gptq" raises an error when Marlin is enabled.

[DreamTeamWangbowen]

@DreamTeamWangbowen IIUC, Marlin is not a quantization method. It's a fast kernel implementation for GPTQ.

I've updated the PR so that it fixes the bug that quantization="gptq" raises an error when Marlin is enabled.

Yeath, you are right, thank you very much. :)