diff --git a/docs/source/assets/deployment/chatbox-chat.png b/docs/source/assets/deployment/chatbox-chat.png new file mode 100644 index 000000000000..b1718cb50471 Binary files /dev/null and b/docs/source/assets/deployment/chatbox-chat.png differ diff --git a/docs/source/assets/deployment/chatbox-settings.png b/docs/source/assets/deployment/chatbox-settings.png new file mode 100644 index 000000000000..a8e3d7b2894c Binary files /dev/null and b/docs/source/assets/deployment/chatbox-settings.png differ diff --git a/docs/source/deployment/frameworks/chatbox.md b/docs/source/deployment/frameworks/chatbox.md new file mode 100644 index 000000000000..e62f4647150f --- /dev/null +++ b/docs/source/deployment/frameworks/chatbox.md @@ -0,0 +1,36 @@ +(deployment-chatbox)= + +# Chatbox + +[Chatbox](https://github.com/chatboxai/chatbox) is a desktop client for LLMs, available on Windows, Mac, Linux. + +It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. + +## Prerequisites + +- Setup vLLM environment + +## Deploy + +- Start the vLLM server with the supported chat completion model, e.g. + +```console +vllm serve qwen/Qwen1.5-0.5B-Chat +``` + +- Download and install [Chatbox desktop](https://chatboxai.app/en#download). + +- On the bottom left of settings, Add Custom Provider + - API Mode: `OpenAI API Compatible` + - Name: vllm + - API Host: `http://{vllm server host}:{vllm server port}/v1` + - API Path: `/chat/completions` + - Model: `qwen/Qwen1.5-0.5B-Chat` + +:::{image} /assets/deployment/chatbox-settings.png +::: + +- Go to `Just chat`, and start to chat: + +:::{image} /assets/deployment/chatbox-chat.png +::: diff --git a/docs/source/deployment/frameworks/index.md b/docs/source/deployment/frameworks/index.md index d1c058eafa4c..aa3394c377d5 100644 --- a/docs/source/deployment/frameworks/index.md +++ b/docs/source/deployment/frameworks/index.md @@ -6,6 +6,7 @@ anything-llm bentoml cerebrium +chatbox dstack helm lws