[Bug]: Support Falcon Mamba

[hahmad2008]

Your current environment

Does VLLM support Falcon Mamba models? if not, when it will be supported

🐛 Describe the bug

Does VLLM support Falcon Mamba models? if not, when it will be supported

[robertgshaw2-redhat]

cc @tlrmchlsmth

[Pinkypoopoo]

Unsubscribe

…

On Wed, 14 Aug, 2024, 1:37 am Robert Shaw, ***@***.***> wrote: cc @tlrmchlsmth <https://github.com/tlrmchlsmth> — Reply to this email directly, view it on GitHub <#7478 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCU643RS6ZZAY5MXB4JRVXLZRJRR3AVCNFSM6AAAAABMO6JHLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXGAZTONBTHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tlrmchlsmth]

Hey @hahmad2008,

No, vLLM doesn't support Falcon Mamba yet. I have a work-in-progress PR (#6484) to add support for Mamba. I'll look into supporting FalconMamba as well.

[hahmad2008]

Thank @tlrmchlsmth. Do you have any idea when this PR will be merged?

[hahmad2008]

@tlrmchlsmth Do you have any idea when this PR will be merged?

[tlrmchlsmth]

Hi @hahmad2008, I’ve been prioritizing that PR over the last couple of days and I think will land it later this week. Do note that I’m not planning to add FalconMamba to #6484, but if it’s similar enough to Mamba or Mamba2, there will be a fast follow PR for support

[hahmad2008]

Thank you @tlrmchlsmth. I can only load and apply inference from it using transformers version 4.45.0.dev0 which is still not released. I installed it using:

pip install -U git+https://github.com/huggingface/transformers.git 

So do you think your PR handle this?

[tlrmchlsmth]

@hahmad2008 sorry, haven't gotten a chance to look at FalconMamba yet -- If transformers 4.45 is needed, then I'll likely I'll wait for that release.

[hahmad2008]

@tlrmchlsmth Thanks! seems it will be released next week.
huggingface/transformers#33236 (comment)

[hassanraha]

i think now it is released. @tlrmchlsmth please update on this.

[tlrmchlsmth]

I'll be landing Mamba soon, possibly today. Sorry for the delay on that one.

FalconMamba looks to be quite close to Mamba, so should be fairly easy to support. I'll try to get to it when I have some spare cycles but can't promise that I'll prioritize it. If somebody else wants to work on FalconMamba before I get to it, I would be happy to shepherd it

[dhiaEddineRhaiem]

Hey Guys,
i managed to add support for FalconMamba inside vllm ,
here is the link: #9325
Please feel free to check that out.

[hassanraha]

Thank you. I will try deploying it.