Stuck in the 'Starting' process

[lxsyz]

Describe the bug
It's stuck in the 'Starting' process.
sudo RUST_BACKTRACE=full ./tabby serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda --port 5050 ⠸ 176.385 s Starting...

Information about your version
tabby 0.14.0
And I tried 0.15.0. It also has the same issue.

Information about your GPU
NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2
NVIDIA A800-SXM4-80GB

Additional context
I just run the command in a docker container. And I have restarted many times.

[wsxiaoys]

Hi - could you try adding RUST_LOG=debug and paste logs here?

[lxsyz]

@wsxiaoys thanks for your reply

2024-08-19T11:03:30.191483Z DEBUG reqwest::connect: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.4/src/connect.rs:497: starting new connection: https://raw.githubusercontent.com/ 2024-08-19T11:03:30.191631Z DEBUG hyper_util::client::legacy::connect::dns: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/dns.rs:122: resolving host="raw.githubusercontent.com" 2024-08-19T11:03:30.195673Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to 185.199.109.133:443 2024-08-19T11:03:30.497121Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to [2606:50c0:8003::154]:443 2024-08-19T11:03:30.497164Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to [2606:50c0:8002::154]:443 2024-08-19T11:03:30.497174Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to [2606:50c0:8001::154]:443 2024-08-19T11:03:30.497183Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:631: connecting to [2606:50c0:8000::154]:443 2024-08-19T11:03:32.427170Z DEBUG hyper_util::client::legacy::connect::http: /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.5/src/client/legacy/connect/http.rs:634: connected to 185.199.109.133:443

[wsxiaoys]

From the error log, it seems your network isn't able to access https://raw.githubusercontent.com/

[SachiaLanlus]

I encounter the same issue.
I am using Docker to run Tabby.
And there is a weird thing that the command displayed in nvtop is /opt/tabby/bin/llama-server -m /data/models/TabbyML/Nomic-Embed-Text/ggml/model.gguf --cont-batching --port 30888 -np 1 --log-disable --ctx-size 4096 -ngl 9999 --embedding --ubatch-size 4096.
But what I configured in compose is serve --model TabbyML/DeepseekCoder-6.7B --device cuda.

[SachiaLanlus]

The latest version of image that work is 0.13.0.
0.14.0, 0.15.0 and latest do not work.

[SachiaLanlus]

Looks like this is the same issue as in #2719

[SachiaLanlus]

I remove all the data and rebuild it. And it works, but it take long time (~10 minute) to start up.

In the nvtop it shows serve command with embedding model first, and it take around 10 minutes to do some setup I think.
During that time it take ~8% of CPU (i5-13500 with only 6 P-Core) consistently.
When it had done the iniitilization, then another command showed up in nvtop which is the command that I specify in compose.yml.
I take 10 minutes every time. And I suspect the reason is that the embedding model only using single thread.