ENH: make deepseek_vl support streaming output

[Minamiyama]

QQ202458-102123.mp4

[sunhaha123]

Looks great! Could this model run on 24gb gpu?

[sunhaha123]

Looks great! Could this model run on 24gb gpu?

seems fp16 works.
using:vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained( # type: ignore self.model_path, trust_remote_code=True, device_map=self._device, low_cpu_mem_usage=True, torch_dtype=torch.float16 )

[Minamiyama]

Looks great! Could this model run on 24gb gpu?

yes, I run it on gtx4090

[sunhaha123]

  File "/home/echo/miniconda3/envs/xinfer/lib/python3.10/site-packages/xinference/model/llm/pytorch/deepseek_vl.py", line 190, in chat
    from ....thirdparty.deepseek_vl.serve.inference import generate
ModuleNotFoundError: [address=0.0.0.0:42331, pid=2580143] No module named 'xinference.thirdparty.deepseek_vl.serve'

I pip install after git pull from main branch. Occur this problem.

[Minamiyama]

  File "/home/echo/miniconda3/envs/xinfer/lib/python3.10/site-packages/xinference/model/llm/pytorch/deepseek_vl.py", line 190, in chat
    from ....thirdparty.deepseek_vl.serve.inference import generate
ModuleNotFoundError: [address=0.0.0.0:42331, pid=2580143] No module named 'xinference.thirdparty.deepseek_vl.serve'

I pip install after git pull from main branch. Occur this problem.

the module codes is in source codes:

maybe you should build from source code, or put the codes in xinference/thirdparty/deepseek_vl to your package path (I solved it by this way)

[sunhaha123]

maybe you should build from source code, or put the codes in xinference/thirdparty/deepseek_vl to your package path (I solved it by this way)

It works. Still feeling strange, I do pip install from source code.

[qinxuye]

Did you update the main code?

[sunhaha123]

Did you update the main code?

git pull origin main, and pip install .[transformers]

[dimitribellini]

Dear Team,
I'm new to "xinference" and I do not understand so well on how to use your inference server and the Deepseek VL model.
Using the docker option I was able to run it and load the module using the engine option "transformer" but as soon a connect using OpenWebUI through the OpenAI API I received the message reported from @Minamiyama
How I can solve it? Your help it's very appreciate :-)
Thanks so much