[Bug] Get connection error when use sglang python module

[kanebay]

$ pip list | grep sglang
sglang 0.3.5

$ pip list | grep flashinfer
flashinfer 0.1.6+cu121torch2.4

$ pip list | grep cu
executing 2.1.0
flashinfer 0.1.6+cu121torch2.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.6.77
nvidia-nvtx-cu12 12.1.105

---

python3 -m sglang.launch_server --model ../deepseek-llm-7b-chat --tp 8 --trust-remote-code

Traceback (most recent call last):
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
yield
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
resp = self._pool.handle_request(req)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
raise exc from None
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
raise exc
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
stream = self._connect(request)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 122, in _connect
stream = self._network_backend.connect_tcp(**kwargs)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
with map_exceptions(exc_map):
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 990, in _request
response = self._client.send(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
response = self._send_handling_auth(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
response = self._send_handling_redirects(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
response = self._send_single_request(request)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
response = transport.handle_request(request)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
with map_httpcore_exceptions():
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/home/puser1/disk2/lei_workspace/llm_test/Deepseek/test_code/sglang.py", line 14, in
response = client.chat.completions.create(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_utils/_utils.py", line 275, in wrapper
return func(*args, **kwargs)
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 829, in create
return self._post(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1277, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 954, in request
return self._request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1014, in _request
return self._retry_request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1092, in _retry_request
return self._request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1014, in _request
return self._retry_request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1092, in _retry_request
return self._request(
File "/home/puser1/miniconda3/envs/sglang/lib/python3.10/site-packages/openai/_base_client.py", line 1024, in _request
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

---

$ python
Python 3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sglang
httpcore.ConnectError: [Errno 111] Connection refused
…………
httpx.ConnectError: [Errno 111] Connection refused
…………
openai.APIConnectionError: Connection error.

Who can provide some ideas？ Thanks very much.