Mixtral-8x7b-32kseqlen

[surak]

Hi,

I just managed to download the whole thing and upload to the inference servers.

Is there support to run Mixtral on FastChat?

Relevant links:
huggingface/transformers#27942
huggingface/transformers@accccdd

[graz68a]

Hope to see Mixtral-8x7b too !

[infwinston]

Latest vllm should support it already. Would be awesome if you guys help us test if vllm_worker.py works for Mixtral-8x7b
https://github.com/vllm-project/vllm

[egortolmachev]

Latest vllm should support it already. Would be awesome if you guys help us test if vllm_worker.py works for Mixtral-8x7b https://github.com/vllm-project/vllm

No, it's dont)))

[surak]

It works - we might need to update the dependency versions of a lot of stuff, as I had to do quite some juggling of "pip install --no-deps" in the middle of the night to get it running, but it works!