unsloth with vllm in 8/4 bits

[quancore]

I have trained qlora model with unsloth and I want to serve with vllm but I did not found a way to serve model in8/4 bits ?

[danielhanchen]

@quancore I'm not sure / unsure if vLLM allows serving in 4 or 8 bits!
16bit yes, but unsure on 4 or 8

[quancore]

@danielhanchen I think it is: vllm-project/vllm#1155

[patleeman]

@danielhanchen I think it is: vllm-project/vllm#1155

Looks like they only support AWQ quantization not via bitsandbytes.

[danielhanchen]

@patleeman Oh ye AWQ is great - I'm assuming you want to quantize it to AWQ?

[quancore]

@patleeman @danielhanchen well yes, maybe we should support AWQ so we can use qlora models with vllm?

[marcelodiaz558]

Hello there. I am also interested in using with VLLM a 8/4 bits model trained with Unsloth. Currently, it works fine with 16 bits but requires too much VRAM. Is there a way to quantize a model trained with Unsloth using AWQ or GPTQ?

[danielhanchen]

Whoops this missed me - yep having an option to convert it to AWQ is interesting

[Louis2B2G]

Whoops this missed me - yep having an option to convert it to AWQ is interesting

That would be amazing - is this a feature you are planning on adding in the near future?

[danielhanchen]

Yep for a future release!

[amir-in-a-cynch]

I'm down to volunteer to work on this, if you're accepting community contributions. (I have to do this for my day job anyway, so it might be nice to contribute to the library.)

[Serega6678]

@amir-in-a-cynch do you plan to do it?

[amir-in-a-cynch]

@amir-in-a-cynch do you plan to do it?

I'll take a stab at it tomorrow and wednesday. Not sure if it'll end up being a clean integration to the API for this library (since it adds a dependency), but at the worst case we should be able to get an example notebook together on how to do it for the docs.

[Serega6678]

@amir-in-a-cynch great, keep me in touch
I don't mind giving you a helping hand if you're stuck at some point

[danielhanchen]

I think vLLM exporting to 8bits is through AWQ - you can also enable float8 support (if your GPU supports it)

[BBiering]

@amir-in-a-cynch @danielhanchen Is there any update on this feature? Would be great to be able to use Unsloth quantized models with vLLM.

[danielhanchen]

Actually I think vLLM added 4bit quants - I need to check it out - I'll make some script fro this!

[frei-x]

unsloth AttributeError: Model Qwen2ForCausalLM does not support BitsAndBytes quantization yet.

[danielhanchen]

@frei-x Oh it should function now hopefully? Please update Unsloth! Sorry on the delay as well!

pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

[nandagopal1992]

@danielhanchen does this mean , the latest version has support for vllm with 4 / 8 bits?

Btw amazing work here :)

[danielhanchen]

@nandagopal1992 I'm pretty certain vLLM can load 4 bit bitsandbytes modules now

[shimmyshimmer]

Now supported! :) Let us know if you still have any issues