[Usage]: using open-webui with vLLM inference engine instead of ollama

[wolfgangsmdt]

Your current environment

No need for the environment

How would you like to use vllm

I want to run inference using openwebui (or something similar) using vLLM as a backend instead of ollama.
I already launched the vLLM openai server api and open-webai. However, it is not working.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[NicolasDrapier]

Hi @wolfgangsmdt

Here is the command I use to run Open-Webui with vLLM:

docker run --name=open-webui \
--env=ENABLE_RAG_WEB_SEARCH=true \
--env=RAG_WEB_SEARCH_ENGINE=duckduckgo \
--env=IMAGE_GENERATION_ENGINE=comfyui \
--env=COMFYUI_BASE_URL=http://<your-ip>:<comfy-port> \
--env=ENABLE_OLLAMA_API=false \
--env=OPENAI_API_KEY=aaaaa \
--env=ENABLE_IMAGE_GENERATION=true \
--env=IMAGE_SIZE=1024x1024 \
--env='IMAGE_GENERATION_MODEL=Stable Diffusion 3 - Medium' \
--env=WHISPER_MODEL=large \
--env=OPENAI_API_BASE_URL=http://<your-ip>:<vllm-port>/v1 \
--volume=/data/open-webui-volumes:/app/backend/data \
-p 3000:8080 \
--restart=always \
ghcr.io/open-webui/open-webui:main

It works like a charm.

[wolfgangsmdt]

Hello @NicolasDrapier ,

Thank you very much for the reply.
I just need little more help.

To start vLLM server I used:

vllm serve TinyLlama/TinyLlama_v1.1 --api-key token-abc123

To start open-webui I used your suggestion:

docker run -d -p 3000:8080 \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
--env=OPENAI_API_BASE_URL=http://localhost:8000/v1 \
--env=OPENAI_API_KEY=token-abc123 \
--env=ENABLE_OLLAMA_API=false \
--env=ENABLE_RAG_WEB_SEARCH=true \
--env=RAG_WEB_SEARCH_ENGINE=duckduckgo  \
ghcr.io/open-webui/open-webui:main

When logging in to open-webui I have nothing and I cannot access Tinyllama

I am missing something in the commands?

EDIT:

I am getting Unauthorized from server side when I run start the open-webui:

INFO:     localhost:43052 - "GET /v1 HTTP/1.1" 401 Unauthorized
WARNING:  Invalid HTTP request received.
WARNING:  Invalid HTTP request received.

However, when I am running this command with curl, I am getting OK, which means the server is running perfectly.

curl http://localhost:8000/health

[wolfgangsmdt]

Hello again @sri-fiddler
do you have any idea about this?

[NicolasDrapier]

Hi @wolfgangsmdt,

You are correct when you say you miss something. When you specify localhost in the IP address for OpenAI in the Docker command, it refers to the localhost inside the Docker container, not the host machine.

To resolve this, you have a couple of options:

1. Use the Host's IP Address: Replace localhost with the IP address of your host machine.
2. Change the Network Mode: Modify the Docker command to use the host network mode. This way, the container will share the network with the host machine. You can do this by adding --network host to your Docker run command.

Here is an example of how you can modify your Docker command:

docker run --network host ..... ghcr.io/open-webui/open-webui:main

Or, if you prefer to use the host's IP address:

docker run --env=OPENAI_API_BASE_URL=http://<your_host_ip>:8000/v1  .... ghcr.io/open-webui/open-webui:main

Please try one of these solutions and let me know if it resolves the issue for you

[wolfgangsmdt]

thank you @NicolasDrapier

It worked, after adding --chat-template from here

For example for my model I used the following command with tool_chat_template_llama3.1_json.jinja

vllm serve TinyLlama/TinyLlama_v1.1 --api-key token-abc123 --chat-template /path/to/tool_chat_template_llama3.1_json.jinja

[JohnConnor123]

same problem. Advice doesn't help

[wolfgangsmdt]

same problem. Advice doesn't help

@JohnConnor123 what doesn't help?

did you put the correct OPENAI_API_BASE_URL and OPENAI_API_KEY make sure that you can curl the vLLM server before to be sure that the vLLM server is running correctly.

curl localhost:8000/health

# OR try using your ip instead of localhost
curl <your-ipv4-add>:8000/health

[NicolasDrapier]

Hi @JohnConnor123,

What is your setup and what are the commands you wrote? Without it, I don't think someone will be able to help you

[JohnConnor123]

same problem. Advice doesn't help

@JohnConnor123 what doesn't help?

did you put the correct OPENAI_API_BASE_URL and OPENAI_API_KEY make sure that you can curl the vLLM server before to be sure that the vLLM server is running correctly.

curl localhost:8000/health

# OR try using your ip instead of localhost
curl <your-ipv4-add>:8000/health

Yes, of course. The problem was difficulties with the docker network, i.e. open webui "did not see" vllm.

P.s. For future readers who want to know exactly how I solved this problem:
You need to add --net host and remove port mapping. This must be done ON BOTH containers.

[JohnConnor123]

Hi @JohnConnor123,

What is your setup and what are the commands you wrote? Without it, I don't think someone will be able to help you

The problem was difficulties with the docker network, i.e. open webui "did not see" vllm. I fixed it :)

[markbaumgarten]

Does vllm work with llama3.3, or only with huggingface models?

[JohnConnor123]

Does vllm work with llama3.3, or only with huggingface models?

llama3.3 exists in huggingface hub. You need to just pass hf_token to vllm docker container. Here is example:
--env "HUGGING_FACE_HUB_TOKEN=<your hf token>"

[markbaumgarten]

Does vllm work with llama3.3, or only with huggingface models?

llama3.3 exists in huggingface hub. You need to just pass hf_token to vllm docker container. Here is example: --env "HUGGING_FACE_HUB_TOKEN=<your hf token>"

Trying to download this model - takes up several hundreds GB of disk space - and in the comments I see that it only supports very few languages....

https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

Lets say i can find disk space somewhere and download it - would I be able to ask it to translate Swedish to English for me - like i can with the Ollama version?