Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: see connection to gpu node timeout issue when initializing ray vllm multi-node serving #13052

Open
1 task done
meqiangxu opened this issue Feb 10, 2025 · 8 comments
Open
1 task done

[Bug]: see connection to gpu node timeout issue when initializing ray vllm multi-node serving #13052

meqiangxu opened this issue Feb 10, 2025 · 8 comments
Labels
bug Something isn't working ray anything related with ray

Comments

@meqiangxu
Copy link

Your current environment

The output of `python collect_env.py` 
INFO 02-10 11:29:04 __init__.py:183] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.16 | packaged by conda-forge | (main, Dec  5 2024, 14:16:10) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.1.119-129.201.amzn2023.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L4
Nvidia driver version: 560.35.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               8
On-line CPU(s) list:                  0-7
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R13 Processor
CPU family:                           25
Model:                                1
Thread(s) per core:                   2
Core(s) per socket:                   4
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             5300.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            128 KiB (4 instances)
L1i cache:                            128 KiB (4 instances)
L2 cache:                             2 MiB (4 instances)
L3 cache:                             16 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-7
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] onnx==1.17.0
[pip3] onnxruntime-gpu==1.20.1
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.0
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-ml-py              12.570.86                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.1                   pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.47.0                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-7	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=GPU-57e82351-9234-210d-8b01-ca2a82b5c223
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.1.1
LD_LIBRARY_PATH=/home/ray/anaconda3/lib/python3.10/site-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I am setting up serving Meta-Llama-3.1-70B-Instruct-GPTQ-INT4 model for inference using two g6.2xlarge instances (each node has one 1gpu). I created placement group like this:
Placement Group Bundles: [{'CPU': 1.0}, {'GPU': 1.0, 'CPU': 1.0, 'memory': 9000000000.0}, {'memory': 9000000000.0, 'CPU': 1.0, 'GPU': 1.0}], and configured it to the serve deployment actor.

            infer_ray_actor_options = {
                "num_gpus": 0,
                "num_cpus": 1,
            }
            handle = serve.run(InferActor.options(ray_actor_options=infer_ray_actor_options,
                                                placement_group_bundles=placement_group_bundles,
                                                placement_group_strategy=placement_group_strategy,
                                                max_ongoing_requests=512,
                                                autoscaling_config=autoscale_config)
                             .bind(model_cache_deployment),
                             route_prefix=None, name=deployment_name)

The AsyncEngineArgs is using tensor_parallel_size=1, pipeline_parallel_size=2. Other args can be seen in the logs below. But I can also attach relevant python code.

Errors

:job_id:01000000
:actor_name:ServeReplica:hugging-quants2-Meta-Llama-3.1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor
INFO 2025-02-10 09:26:45,197 hugging-quants2-Meta-Llama-3.1-70B-Instruct-GPTQ-INT4@infer_actor_InferActor 9em0j31v -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7f5b5b82f2b0>.
Loading model for inference: hugging-quants2/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4
Loading model: hugging-quants2/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4
Placement Group Bundles: [{'CPU': 1.0}, {'GPU': 1.0, 'CPU': 1.0, 'memory': 9000000000.0}, {'memory': 9000000000.0, 'CPU': 1.0, 'GPU': 1.0}]
GPUs per node: [0, 1.0, 1.0], Total GPUs: 2.0
Configured parallel sizes - Tensor: 1, Pipeline: 2 (using 2/2.0 GPUs)
Engine args: AsyncEngineArgs(model='/tmp/tmpojalf_6s', served_model_name=None, tokenizer='/tmp/tmpojalf_6s', task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format=<class 'actors.vllm_loader.RayObjectLoader'>, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', seed=0, max_model_len=1024, distributed_executor_backend='ray', pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, block_size=None, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=True, swap_space=1, cpu_offload_gb=0, gpu_memory_utilization=0.98, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, fully_sharded_loras=False, lora_extra_vocab_size=256, long_lora_scaling_factors=None, lora_dtype='auto', max_cpu_loras=None, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config=<actors.ray_get_generator.RayGetGenerator object at 0x7f5b5846bb80>, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, speculative_model=None, speculative_model_quantization=None, speculative_draft_tensor_parallel_size=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, qlora_adapter_name_or_path=None, disable_logprobs_during_spec_decoding=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', kv_transfer_config=None, generation_config=None, enable_sleep_mode=False, calculate_kv_scales=None, disable_log_requests=False)
SIGTERM handler is not set because current thread is not the main thread.
Using address pfcp-devczpmd0eq-ray-raycluster-ljvdj-head-svc.pfcp-devczpmd0eq-ray.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
Connecting to existing Ray cluster at address: pfcp-devczpmd0eq-ray-raycluster-ljvdj-head-svc.pfcp-devczpmd0eq-ray.svc.cluster.local:6379...
Calling ray.init() again after it has already been called.
[E210 09:46:29.913158643 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (248.2.20.163, 54337).
[W210 09:46:29.913465429 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host 248.2.20.163:54337 - retrying (try=0, timeout=600000ms, delay=54518ms): The client socket has timed out after 600000ms while trying to connect to (248.2.20.163, 54337).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5b40898446 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f5b109ad4c6 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f5b153f6d95 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x6029f36 (0x7f5b153f6f36 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f5b153f73a4 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f5b153b5016 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f5b153b7f7c in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xd9acdd (0x7f5b2b53acdd in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4cb474 (0x7f5b2ac6b474 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x13b6f6 (0x5599cf9b06f6 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #10: _PyObject_MakeTpCall + 0x2d3 (0x5599cf9a9a03 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #11: <unknown function> + 0x147296 (0x5599cf9bc296 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #12: PyVectorcall_Call + 0xc9 (0x5599cf9bccc9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #13: <unknown function> + 0x144ff0 (0x5599cf9b9ff0 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #14: <unknown function> + 0x134d0b (0x5599cf9a9d0b in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #15: <unknown function> + 0x4c9ccb (0x7f5b2ac69ccb in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x2d3 (0x5599cf9a9a03 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #17: _PyEval_EvalFrameDefault + 0x5553 (0x5599cf9a5ff3 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #18: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #19: _PyEval_EvalFrameDefault + 0x309 (0x5599cf9a0da9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #20: <unknown function> + 0x1aaa20 (0x5599cfa1fa20 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #21: <unknown function> + 0x13bd43 (0x5599cf9b0d43 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #22: _PyEval_EvalFrameDefault + 0x309 (0x5599cf9a0da9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #23: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #24: PyObject_Call + 0xbc (0x5599cf9bc94c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #25: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #26: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #27: PyObject_Call + 0xbc (0x5599cf9bc94c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #28: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #29: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #30: _PyEval_EvalFrameDefault + 0x1341 (0x5599cf9a1de1 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #31: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #32: _PyEval_EvalFrameDefault + 0x309 (0x5599cf9a0da9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #33: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #34: _PyEval_EvalFrameDefault + 0x309 (0x5599cf9a0da9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #35: <unknown function> + 0x147234 (0x5599cf9bc234 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #36: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #37: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #38: _PyEval_EvalFrameDefault + 0x309 (0x5599cf9a0da9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #39: <unknown function> + 0x147108 (0x5599cf9bc108 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #40: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #41: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #42: _PyEval_EvalFrameDefault + 0x6fd (0x5599cf9a119d in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #43: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #44: _PyEval_EvalFrameDefault + 0x6fd (0x5599cf9a119d in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #45: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #46: _PyEval_EvalFrameDefault + 0x6fd (0x5599cf9a119d in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #47: <unknown function> + 0x146fb2 (0x5599cf9bbfb2 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #48: PyObject_Call + 0xbc (0x5599cf9bc94c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #49: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #50: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #51: _PyObject_FastCallDictTstate + 0x187 (0x5599cf9a8f47 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #52: <unknown function> + 0x144ad9 (0x5599cf9b9ad9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #53: _PyObject_MakeTpCall + 0x2eb (0x5599cf9a9a1b in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #54: _PyEval_EvalFrameDefault + 0x5553 (0x5599cf9a5ff3 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #55: <unknown function> + 0x146fb2 (0x5599cf9bbfb2 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #56: PyObject_Call + 0xbc (0x5599cf9bc94c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #57: _PyEval_EvalFrameDefault + 0x2c3e (0x5599cf9a36de in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #58: _PyFunction_Vectorcall + 0x6c (0x5599cf9b0b7c in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #59: _PyObject_FastCallDictTstate + 0x187 (0x5599cf9a8f47 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #60: <unknown function> + 0x144ad9 (0x5599cf9b9ad9 in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #61: <unknown function> + 0x134d0b (0x5599cf9a9d0b in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)
frame #62: PyObject_Call + 0x20f (0x5599cf9bca9f in ray::1-70B-Instruct-GPTQ-INT4@infer_actor:InferActor.initialize_and_get_metadata)

I tried using debug script on one gpu node.

NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=1 --rdzv_backend=c10d --rdzv_endpoint=248.2.20.163 test.py

It reports the same error:

(base) ray@pfcp-devczpmd0eq-ray-raycluster-ljvdj-gpu-group-worker-6v6ch:/tmp$ NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=1 --rdzv_backend=c10d --rdzv_endpoint=248.2.20.163 test.py
[E210 09:54:57.124956310 socket.cpp:1011] [c10d] The client socket has timed out after 60000ms while trying to connect to (248.2.20.163, 29400).
[W210 09:54:57.125243957 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host 248.2.20.163:29400 - retrying (try=0, timeout=60000ms, delay=15196ms): The client socket has timed out after 60000ms while trying to connect to (248.2.20.163, 29400).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe591ab9446 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7fe57cd4d4c6 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7fe581796d95 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x6029f36 (0x7fe581796f36 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7fe5817973a4 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7fe581755016 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fe581757f7c in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xd9acdd (0x7fe59113acdd in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4cb474 (0x7fe59086b474 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x13b6f6 (0x5610a9b9b6f6 in /home/ray/anaconda3/bin/python3.10)
frame #10: _PyObject_MakeTpCall + 0x2d3 (0x5610a9b94a03 in /home/ray/anaconda3/bin/python3.10)
frame #11: <unknown function> + 0x147296 (0x5610a9ba7296 in /home/ray/anaconda3/bin/python3.10)
frame #12: PyVectorcall_Call + 0xc9 (0x5610a9ba7cc9 in /home/ray/anaconda3/bin/python3.10)
frame #13: <unknown function> + 0x144ff0 (0x5610a9ba4ff0 in /home/ray/anaconda3/bin/python3.10)
frame #14: <unknown function> + 0x134d0b (0x5610a9b94d0b in /home/ray/anaconda3/bin/python3.10)
frame #15: <unknown function> + 0x4c9ccb (0x7fe590869ccb in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x2d3 (0x5610a9b94a03 in /home/ray/anaconda3/bin/python3.10)
frame #17: _PyEval_EvalFrameDefault + 0x5553 (0x5610a9b90ff3 in /home/ray/anaconda3/bin/python3.10)
frame #18: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #19: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #20: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #21: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #22: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #23: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #24: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #25: _PyEval_EvalFrameDefault + 0x6fd (0x5610a9b8c19d in /home/ray/anaconda3/bin/python3.10)
frame #26: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #27: _PyEval_EvalFrameDefault + 0x49b7 (0x5610a9b90457 in /home/ray/anaconda3/bin/python3.10)
frame #28: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #29: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5610a9b93e90 in /home/ray/anaconda3/bin/python3.10)
frame #31: _PyObject_Call_Prepend + 0x69 (0x5610a9ba55b9 in /home/ray/anaconda3/bin/python3.10)
frame #32: <unknown function> + 0x205f29 (0x5610a9c65f29 in /home/ray/anaconda3/bin/python3.10)
frame #33: PyObject_Call + 0x20f (0x5610a9ba7a9f in /home/ray/anaconda3/bin/python3.10)
frame #34: _PyEval_EvalFrameDefault + 0x2c3e (0x5610a9b8e6de in /home/ray/anaconda3/bin/python3.10)
frame #35: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #36: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #37: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #38: _PyEval_EvalFrameDefault + 0x2c3e (0x5610a9b8e6de in /home/ray/anaconda3/bin/python3.10)
frame #39: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #40: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #41: <unknown function> + 0x1cca8c (0x5610a9c2ca8c in /home/ray/anaconda3/bin/python3.10)
frame #42: PyEval_EvalCode + 0x87 (0x5610a9c2c9d7 in /home/ray/anaconda3/bin/python3.10)
frame #43: <unknown function> + 0x1fceba (0x5610a9c5ceba in /home/ray/anaconda3/bin/python3.10)
frame #44: <unknown function> + 0x1f8343 (0x5610a9c58343 in /home/ray/anaconda3/bin/python3.10)
frame #45: <unknown function> + 0x97098 (0x5610a9af7098 in /home/ray/anaconda3/bin/python3.10)
frame #46: _PyRun_SimpleFileObject + 0x1bd (0x5610a9c52b7d in /home/ray/anaconda3/bin/python3.10)
frame #47: _PyRun_AnyFileObject + 0x44 (0x5610a9c52714 in /home/ray/anaconda3/bin/python3.10)
frame #48: Py_RunMain + 0x31b (0x5610a9c4fa7b in /home/ray/anaconda3/bin/python3.10)
frame #49: Py_BytesMain + 0x37 (0x5610a9c20117 in /home/ray/anaconda3/bin/python3.10)
frame #50: <unknown function> + 0x29d90 (0x7fe592793d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #51: __libc_start_main + 0x80 (0x7fe592793e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #52: <unknown function> + 0x1c002e (0x5610a9c2002e in /home/ray/anaconda3/bin/python3.10)

[E210 09:55:47.152519804 socket.cpp:1011] [c10d] The client socket has timed out after 60000ms while trying to connect to (248.2.20.163, 29400).
[E210 09:55:47.152633429 TCPStore.cpp:346] [c10d] TCP client failed to connect/validate to host 248.2.20.163:29400 - timed out (try=1, timeout=60000ms): The client socket has timed out after 60000ms while trying to connect to (248.2.20.163, 29400).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe591ab9446 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7fe57cd4d4c6 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7fe581796d95 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x6029f36 (0x7fe581796f36 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7fe5817973a4 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7fe581755016 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fe581757f7c in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xd9acdd (0x7fe59113acdd in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4cb474 (0x7fe59086b474 in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x13b6f6 (0x5610a9b9b6f6 in /home/ray/anaconda3/bin/python3.10)
frame #10: _PyObject_MakeTpCall + 0x2d3 (0x5610a9b94a03 in /home/ray/anaconda3/bin/python3.10)
frame #11: <unknown function> + 0x147296 (0x5610a9ba7296 in /home/ray/anaconda3/bin/python3.10)
frame #12: PyVectorcall_Call + 0xc9 (0x5610a9ba7cc9 in /home/ray/anaconda3/bin/python3.10)
frame #13: <unknown function> + 0x144ff0 (0x5610a9ba4ff0 in /home/ray/anaconda3/bin/python3.10)
frame #14: <unknown function> + 0x134d0b (0x5610a9b94d0b in /home/ray/anaconda3/bin/python3.10)
frame #15: <unknown function> + 0x4c9ccb (0x7fe590869ccb in /home/ray/anaconda3/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x2d3 (0x5610a9b94a03 in /home/ray/anaconda3/bin/python3.10)
frame #17: _PyEval_EvalFrameDefault + 0x5553 (0x5610a9b90ff3 in /home/ray/anaconda3/bin/python3.10)
frame #18: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #19: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #20: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #21: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #22: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #23: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #24: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #25: _PyEval_EvalFrameDefault + 0x6fd (0x5610a9b8c19d in /home/ray/anaconda3/bin/python3.10)
frame #26: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #27: _PyEval_EvalFrameDefault + 0x49b7 (0x5610a9b90457 in /home/ray/anaconda3/bin/python3.10)
frame #28: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #29: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5610a9b93e90 in /home/ray/anaconda3/bin/python3.10)
frame #31: _PyObject_Call_Prepend + 0x69 (0x5610a9ba55b9 in /home/ray/anaconda3/bin/python3.10)
frame #32: <unknown function> + 0x205f29 (0x5610a9c65f29 in /home/ray/anaconda3/bin/python3.10)
frame #33: PyObject_Call + 0x20f (0x5610a9ba7a9f in /home/ray/anaconda3/bin/python3.10)
frame #34: _PyEval_EvalFrameDefault + 0x2c3e (0x5610a9b8e6de in /home/ray/anaconda3/bin/python3.10)
frame #35: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #36: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #37: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #38: _PyEval_EvalFrameDefault + 0x2c3e (0x5610a9b8e6de in /home/ray/anaconda3/bin/python3.10)
frame #39: _PyFunction_Vectorcall + 0x6c (0x5610a9b9bb7c in /home/ray/anaconda3/bin/python3.10)
frame #40: _PyEval_EvalFrameDefault + 0x309 (0x5610a9b8bda9 in /home/ray/anaconda3/bin/python3.10)
frame #41: <unknown function> + 0x1cca8c (0x5610a9c2ca8c in /home/ray/anaconda3/bin/python3.10)
frame #42: PyEval_EvalCode + 0x87 (0x5610a9c2c9d7 in /home/ray/anaconda3/bin/python3.10)
frame #43: <unknown function> + 0x1fceba (0x5610a9c5ceba in /home/ray/anaconda3/bin/python3.10)
frame #44: <unknown function> + 0x1f8343 (0x5610a9c58343 in /home/ray/anaconda3/bin/python3.10)
frame #45: <unknown function> + 0x97098 (0x5610a9af7098 in /home/ray/anaconda3/bin/python3.10)
frame #46: _PyRun_SimpleFileObject + 0x1bd (0x5610a9c52b7d in /home/ray/anaconda3/bin/python3.10)
frame #47: _PyRun_AnyFileObject + 0x44 (0x5610a9c52714 in /home/ray/anaconda3/bin/python3.10)
frame #48: Py_RunMain + 0x31b (0x5610a9c4fa7b in /home/ray/anaconda3/bin/python3.10)
frame #49: Py_BytesMain + 0x37 (0x5610a9c20117 in /home/ray/anaconda3/bin/python3.10)
frame #50: <unknown function> + 0x29d90 (0x7fe592793d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #51: __libc_start_main + 0x80 (0x7fe592793e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #52: <unknown function> + 0x1c002e (0x5610a9c2002e in /home/ray/anaconda3/bin/python3.10)

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 159, in _create_tcp_store
    store = TCPStore(
torch.distributed.DistNetworkError: The client socket has timed out after 60000ms while trying to connect to (248.2.20.163, 29400).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ray/anaconda3/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 241, in launch_agent
    rdzv_handler=rdzv_registry.get_rendezvous_handler(rdzv_parameters),
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/registry.py", line 71, in get_rendezvous_handler
    return handler_registry.create_handler(params)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/api.py", line 365, in create_handler
    handler = creator(params)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/registry.py", line 41, in _create_c10d_handler
    backend, store = create_backend(params)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 257, in create_backend
    store = _create_tcp_store(params)
  File "/home/ray/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 183, in _create_tcp_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@meqiangxu meqiangxu added the bug Something isn't working label Feb 10, 2025
@meqiangxu meqiangxu changed the title [Bug]: see connection to gpu node timeout issue using ray vllm multi-node serving [Bug]: see connection to gpu node timeout issue when initializing ray vllm multi-node serving Feb 10, 2025
@meqiangxu
Copy link
Author

Note: it is ray serving in k8

@mrjiangguoqing
Copy link

mrjiangguoqing commented Feb 11, 2025 •
edited
Loading

confirm this bug.
I also run into this problem,the phenomenon is same as above mentioned,the env is rayservice with kube.

[E210 09:46:29.913158643 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to . [W210 09:46:29.913465429 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host - retrying (try=0, timeout=600000ms, delay=54518ms): The client socket has timed out after 600000ms while trying to connect to (host, port).
，
but I test two host with
NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=1 --rdzv_backend=c10d --rdzv_endpoint=10.0.0.252 test.py ,
all check are passed.

ray version 2.41, with vllm version 0.7.2 and 0.7.1 both find the bug.

I also checked the container network,the target pod try to connect,the pod's other port is available.but not the destination port,so I think it doesn't the container network problem.

from the log message I found

workers: [RayWorkerMetaData(worker=Actor(RayWorkerWrapper, ca90c655e70ed712edde745a01000000), created_rank=1, adjusted_rank=-1, ip='192.168.221.145')]

INFO 02-07 00:58:44 utils.py:513] Port 49386 is already in use, trying port 49387

DEBUG 02-08 01:56:15 parallel_state.py:951] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://192.168.245.138:49386 backend=nccl

the different rank init address could different compare with rank 0.

I also try to set the VLLM_HOST_IP and VLLM_PORT,the problem still exist.
I guess maybe the rank init address wrong or something I missed,we need more knowledge to dig it deeper.

@mrjiangguoqing
Copy link

mrjiangguoqing commented Feb 11, 2025 •
edited
Loading

@youkaichao @Jeffwan @houseroad
If you could share some insights, that would be very helpful. I could make more test.

@meqiangxu
Copy link
Author

I also tested with one node 4 gpus, instead of two node each 1 gpu. The error persists.
My ray version is 2.42.0. vLLM version 0.7.0.

@mrjiangguoqing
Copy link

update，back to version 0.6.4.post1 solved the problem.I found in the my env,the 0.7.x PyTorch NCCL test could pass but vllm nccl test fail. back to 0.6.4.post1 success, so I guess the problem is new version bring some communication problems.
below is check faild log in two containers.
llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13423 [0] NCCL INFO Connected all rings llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13423 [0] NCCL INFO Connected all trees llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13423 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13423 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13423 [0] NCCL INFO ncclCommInitRank comm 0x3c020f80 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 70 commId 0x86078063f5a85ee6 - Init COMPLETE [rank0]: Traceback (most recent call last): [rank0]: File "/home/ray/test.py", line 43, in <module> [rank0]: assert value == world_size, f"Expected {world_size}, got {value}" [rank0]: AssertionError: Expected 2, got 1.0 [rank0]:[W211 23:28:33.893717234 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13437 [0] NCCL INFO [Service thread] Connection closed by localRank 0 llama-3-8b-raycluster-zqvtd-gpu-group-worker-cl2ns:13423:13471 [0] NCCL

@meqiangxu maye this also works for you.

@meqiangxu
Copy link
Author

Thanks! Yesterday we used vllm 0.6.5 which could resolve the issue too.

@meqiangxu
Copy link
Author

youkaichao @Jeffwan @houseroad

Do we want this issue be prioritized? Seems it is regression.

@ruisearch42 ruisearch42 added the ray anything related with ray label Feb 19, 2025
@youkaichao
Copy link
Member

update，back to version 0.6.4.post1 solved the problem.I found in the my env,the 0.7.x PyTorch NCCL test could pass but vllm nccl test fail. back to 0.6.4.post1 success, so I guess the problem is new version bring some communication problems.
below is check faild log in two containers.

for this, it might be that the test script is obsolete. i just updated it yesterday in #13487

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working ray anything related with ray
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@youkaichao @meqiangxu @ruisearch42 @mrjiangguoqing