[Serve.LLM] Failed to launch disaggregated prefiller & decoder worker when enabled pipeline_parallel_size =2

[sheperdh]

When I use below serveConfig to launch vllm

serveConfigV2: |
applications:
- name: qianwen2
import_path: ray.llm._internal.serve.deployments.prefill_decode_disagg.prefill_decode_disagg:build_app
route_prefix: /
args:
prefill_config:
model_loading_config:
model_id: qianwen2
model_source: /home/ray/models
engine_kwargs:
dtype: auto
device: auto
max_model_len: 8192
pipeline_parallel_size: 2
tensor_parallel_size: 1
max_num_seqs: 40
gpu_memory_utilization: 0.85
trust_remote_code: true
enforce_eager: true
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 3
runtime_env:
env_vars:
VLLM_USE_V1: "1"
decode_config:
model_loading_config:
model_id: qianwen2
model_source: /home/ray/models
engine_kwargs:
dtype: auto
device: auto
max_model_len: 8192
pipeline_parallel_size: 2
tensor_parallel_size: 1
max_num_seqs: 40
gpu_memory_utilization: 0.85
trust_remote_code: true
enforce_eager: true
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 3
runtime_env:
env_vars:
VLLM_USE_V1: "1"

I got below error in prefll and decoder:

�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/threading.py", line 982, in run
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m self._target(*self._args, **self._kwargs)
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py", line 469, in _nixl_handshake_listener
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m with zmq_ctx(zmq.ROUTER, path) as sock:
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/contextlib.py", line 137, in enter
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m return next(self.gen)
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m ^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py", line 1097, in zmq_ctx
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m yield make_zmq_socket(ctx=ctx,
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/utils/init.py", line 2601, in make_zmq_socket
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m socket.bind(path)
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/zmq/sugar/socket.py", line 311, in bind
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m super().bind(addr)
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "_zmq.py", line 898, in zmq.backend.cython._zmq.Socket.bind
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
�[36m(RayWorkerWrapper pid=908, ip=10.244.121.4)�[0m zmq.error.ZMQError: Cannot assign requested address (addr='tcp://10.244.124.3:41059')

It seems the address 10.244.124.3:41059 in another ray worker is not accessible from the prefiller/decoder.
From my understanding, when ray starts vllm engine, it needs to initializ nixl connector, so when it initialize nixl worker agent, it needs to talk to remote nixl agent, but because right now ray cluster doesn't know nixl port, so it hasn't exposed the port, which blocks the nixl connector initialization.

[kouroshHakha]

It's not clear from the stack trace, but is the issue that the nixl port used is colliding with something else using the port already? Does this issue happen in PP = 1 ? PP > 1 has another known problem with ray on vllm which we'll be investigating soon, but this stack trace seems different from that error.

For nixl port collision problem I have another PR: #55802

[sheperdh]

@kouroshHakha Thanks for your information.
I double checked the ray worker pod, you are correct, the port has already been used by others. The question is I can't see which process is listening that port from netstat output.
If not specifying PP, I can serve the model successfully.
I will check whether this issue is fixed with your PR.

[sheperdh]

After upgrading the image to 2.49.0.5531bc-py311-cu128, I still see the same error:

�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/threading.py", line 982, in run
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m self._target(*self._args, **self._kwargs)
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py", line 474, in _nixl_handshake_listener
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m with zmq_ctx(zmq.ROUTER, path) as sock:
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/contextlib.py", line 137, in enter
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m return next(self.gen)
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m ^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py", line 1076, in zmq_ctx
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m yield make_zmq_socket(ctx=ctx,
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/utils/init.py", line 2836, in make_zmq_socket
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m socket.bind(path)
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "/home/ray/anaconda3/lib/python3.11/site-packages/zmq/sugar/socket.py", line 311, in bind
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m super().bind(addr)
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "_zmq.py", line 898, in zmq.backend.cython._zmq.Socket.bind
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m zmq.error.ZMQError: Cannot assign requested address (addr='tcp://10.244.175.5:55000')

Here is the log I saw:

DEBUG 08-26 21:52:51 [init.py:40] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-26 21:52:51 [init.py:43] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 08-26 21:52:51 [core.py:71] Initializing a V1 LLM engine (v0.10.0) with config: model='/home/ray/models', speculative_config=None, tokenizer='/home/ray/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qianwen-test, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
INFO 08-26 21:52:51 [ray_utils.py:315] Using the existing placement group
INFO 08-26 21:52:51 [ray_distributed_executor.py:169] use_ray_spmd_worker: True
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:30] No plugins for group vllm.platform_plugins found.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:35] Checking if TPU platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:45] TPU platform is not available because: No module named 'libtpu'
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:52] Checking if CUDA platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:72] Confirmed CUDA platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:100] Checking if ROCm platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:114] ROCm platform is not available because: No module named 'amdsmi'
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:121] Checking if XPU platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:140] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:147] Checking if CPU platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:169] Checking if Neuron platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:52] Checking if CUDA platform is available.
�[36m(pid=607)�[0m DEBUG 08-26 21:52:55 [init.py:72] Confirmed CUDA platform is available.
�[36m(pid=607)�[0m INFO 08-26 21:52:55 [init.py:235] Automatically detected platform cuda.
DEBUG 08-26 21:52:56 [ray_distributed_executor.py:244] workers: [RayWorkerMetaData(worker=Actor(RayWorkerWrapper, 3e5129c528b9c28af26b014f02000000), created_rank=0, adjusted_rank=-1, ip='10.244.175.5'), RayWorkerMetaData(worker=Actor(RayWorkerWrapper, 1eea2250d6915a6ee444ac1e02000000), created_rank=1, adjusted_rank=-1, ip='10.244.176.7')]
DEBUG 08-26 21:52:56 [ray_distributed_executor.py:245] driver_dummy_worker: None
INFO 08-26 21:52:56 [ray_env.py:63] RAY_NON_CARRY_OVER_ENV_VARS from config: set()
INFO 08-26 21:52:56 [ray_env.py:65] Copying the following environment variables to workers: ['VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_LOGGING_LEVEL', 'VLLM_NIXL_SIDE_CHANNEL_PORT', 'VLLM_USE_V1', 'LD_LIBRARY_PATH', 'VLLM_USE_RAY_SPMD_WORKER', 'VLLM_NIXL_SIDE_CHANNEL_HOST', 'VLLM_USE_RAY_COMPILED_DAG', 'VLLM_CPU_KVCACHE_SPACE']
INFO 08-26 21:52:56 [ray_env.py:68] If certain env vars should NOT be copied, add them to /home/ray/.config/vllm/ray_non_carry_over_env_vars.json file
DEBUG 08-26 21:53:01 [utils.py:741] Waiting for 1 local, 0 remote core engine proc(s) to start.
�[36m(RayWorkerWrapper pid=607)�[0m WARNING 08-26 21:52:56 [init.py:949] Overwriting environment variable LD_LIBRARY_PATH from '/usr/local/cuda/lib64:/usr/local/ucx/lib:/usr/local/nixl/lib/x86_64-linux-gnu:/usr/local/nvidia/lib64' to '/home/ray/anaconda3/lib/python3.11/site-packages/cv2/../../lib64:/usr/local/cuda/lib64:/usr/local/ucx/lib:/usr/local/nixl/lib/x86_64-linux-gnu:/usr/local/nvidia/lib64'
�[36m(RayWorkerWrapper pid=607)�[0m DEBUG 08-26 21:52:56 [init.py:38] Available plugins for group vllm.general_plugins:
�[36m(RayWorkerWrapper pid=607)�[0m DEBUG 08-26 21:52:56 [init.py:40] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
�[36m(RayWorkerWrapper pid=607)�[0m DEBUG 08-26 21:52:56 [init.py:43] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
�[36m(autoscaler +9s, ip=10.244.176.8)�[0m Tip: use ray status to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.
�[36m(autoscaler +9s, ip=10.244.176.8)�[0m No available node types can fulfill resource requests {'bundle_group_29526f599b799876ca32462ae38d01000000': 0.001, 'CPU_group_29526f599b799876ca32462ae38d01000000': 1.0}*1. Add suitable node types to this cluster to resolve this issue.
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:58 [decorators.py:139] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:58 [decorators.py:139] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama_eagle3.LlamaModel'>: ['input_ids', 'positions', 'hidden_states']
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:59 [init.py:3051] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fb6ce68aed0>
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:59 [config.py:4871] enabled custom ops: Counter()
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:59 [config.py:4873] disabled custom ops: Counter()
�[36m(RayWorkerWrapper pid=607)�[0m DEBUG 08-26 21:52:59 [parallel_state.py:945] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://10.244.175.5:47587 backend=nccl
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m DEBUG 08-26 21:52:59 [parallel_state.py:996] Detected 2 nodes in the distributed environment
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:52:59 [init.py:1375] Found nccl from library libnccl.so.2
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:52:59 [pynccl.py:70] vLLM is using nccl==2.26.2
�[36m(RayWorkerWrapper pid=469, ip=10.244.176.7)�[0m INFO 08-26 21:53:00 [parallel_state.py:1102] rank 1 in world size 2 is assigned as DP rank 0, PP rank 1, TP rank 0, EP rank 0
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:53:00 [nixl_connector.py:50] NIXL is available
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:53:00 [factory.py:74] Creating v1 connector with name: NixlConnector and engine_id: ccc2d5ce-792d-41f6-aee2-98b736cd851e-10.244.175.5-55000
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:53:00 [nixl_connector.py:356] Initializing NIXL wrapper
�[36m(RayWorkerWrapper pid=607)�[0m INFO 08-26 21:53:00 [nixl_connector.py:357] Initializing NIXL worker ccc2d5ce-792d-41f6-aee2-98b736cd851e-10.244.175.5-55000
�[36m(RayWorkerWrapper pid=607)�[0m Backend UCX was instantiated
�[36m(RayWorkerWrapper pid=607)�[0m Initialized NIXL agent: 57fdf82f-05c5-4d6f-81b0-3dae621b457c

It looks like nixl agent is initialized correctly, but the nixl_connector failed because port conflicts.

[kouroshHakha]

you have to specify NIXL_SIDE_CHANNEL_PORT_BASE in experimental_configs. Are you doing that?

[sheperdh]

Well, I have set "VLLM_NIXL_SIDE_CHANNEL_PORT" to "55000". What NIXL_SIDE_CHANNEL_PORT_BASE will be used?

[sheperdh]

I tried to set NIXL_SIDE_CHANNEL_PORT_BASE to 55000, still the same error.

[kouroshHakha]

Can you share your full llm config ? and exact repro steps?

[sheperdh]

test.yaml
Here is my config from kubectl get.

[forestlee95]

Does Ray currently support using LMcache/NIXL to create a unified kvcache memory pool across the CPU/GPU space of multiple nodes?