Skip to content

[Bug]: vLLM (TP=8) on 235B model triggers "CUDA error: unspecified launch failure" and persistent "ERR!" state in nvidia-smi #27430

New issue
New issue
Open
Open
[Bug]: vLLM (TP=8) on 235B model triggers "CUDA error: unspecified launch failure" and persistent "ERR!" state in nvidia-smi#27430
Labels
bugSomething isn't workingnvidia
@whwangovo

Description

@whwangovo
whwangovo
opened on Oct 23, 2025

Your current environment

The output of python collect_env.py 
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.3 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.1.105
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration :
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version        : 470.199.02
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Address sizes:       46 bits physical, 57 bits virtual
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Vendor ID:           GenuineIntel
Model name:          Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz
CPU family:          6
Model:               106
Thread(s) per core:  2
Core(s) per socket:  32
Socket(s):           2
Stepping:            6
CPU max MHz:         3500.0000
CPU min MHz:         800.0000
BogoMIPS:            5800.00
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities
Virtualization:      VT-x
L1d cache:           3 MiB (64 instances)
L1i cache:           2 MiB (64 instances)
L2 cache:            80 MiB (64 instances)
L3 cache:            96 MiB (2 instances)
NUMA node(s):        2
NUMA node0 CPU(s):   0-31,64-95
NUMA node1 CPU(s):   32-63,96-127

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.1
[pip3] triton==3.4.0
[conda] numpy                                2.2.6            pypi_0           pypi
[conda] nvidia-cublas-cu12                   12.8.4.1         pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12               12.8.90          pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12               12.8.93          pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12             12.8.90          pypi_0           pypi
[conda] nvidia-cudnn-cu12                    9.10.2.21        pypi_0           pypi
[conda] nvidia-cufft-cu12                    11.3.3.83        pypi_0           pypi
[conda] nvidia-cufile-cu12                   1.13.1.3         pypi_0           pypi
[conda] nvidia-curand-cu12                   10.3.9.90        pypi_0           pypi
[conda] nvidia-cusolver-cu12                 11.7.3.90        pypi_0           pypi
[conda] nvidia-cusparse-cu12                 12.5.8.93        pypi_0           pypi
[conda] nvidia-cusparselt-cu12               0.7.1            pypi_0           pypi
[conda] nvidia-nccl-cu12                     2.27.3           pypi_0           pypi
[conda] nvidia-nvjitlink-cu12                12.8.93          pypi_0           pypi
[conda] nvidia-nvtx-cu12                     12.8.90          pypi_0           pypi
[conda] pyzmq                                27.1.0           pypi_0           pypi
[conda] torch                                2.8.0            pypi_0           pypi
[conda] torchaudio                           2.8.0            pypi_0           pypi
[conda] torchvision                          0.23.0           pypi_0           pypi
[conda] transformers                         4.57.1           pypi_0           pypi
[conda] triton                               3.4.0            pypi_0           pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-31,64-95      0
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-31,64-95      0
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-31,64-95      0
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-31,64-95      0
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    32-63,96-127    1
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    32-63,96-127    1
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    32-63,96-127    1
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      32-63,96-127    1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I am running the vLLM API server with a very large model (Qwen3-VL-235B-A22B-Instruct) using 8-way tensor parallelism. The server starts and can successfully process some requests.

However, after a period of operation (or when handling a specific request), the service encounters a catastrophic CUDA error and crashes the entire engine.

The Crash

The failure sequence is as follows:

  1. The first critical error appears from rank4: [rank4]:[E1023 17:16:46.022239569 ProcessGroupNCCL.cpp:2068] ... Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure. This originates from ProcessGroupNCCL.
  2. The API server immediately begins returning 500 Internal Server Error to all incoming requests.
  3. Approximately 60 seconds later, the EngineCore reports a timeout from the shared memory broadcast: shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds.
  4. This is immediately followed by the error: ERROR ... [multiproc_executor.py:154] Worker proc VllmWorker-4 died unexpectedly, shutting down executor.
  5. All other workers then terminate (Parent process exited, terminating worker).
  6. The EngineCore and APIServer log a fatal vllm.v1.engine.exceptions.EngineDeadError.
  7. The process exits uncleanly, leaving nanobind and multiprocessing:resource_tracker leaks.

Critical Side Effect: GPU Hardware State

The most critical part of this bug is that after the vLLM process crashes, the underlying NVIDIA GPUs are left in a "dead" state.

Running nvidia-smi after the crash (while no processes are running) shows a persistent ERR! for the Power/Usage reading on the affected GPUs (GPU 4 and 5 in the attached log), confirming this is a driver-level or hardware-level crash, not just a software exception. The only way to recover the GPUs from this ERR! state is a full system reboot.

My Script

vllm serve checkpoints/Qwen/Qwen3-VL-235B-A22B-Instruct \
  --served-model-name qwenvl \
  --tensor-parallel-size 8 \
  --max-model-len 124000 \
  --gpu-memory-utilization 0.95 \
  --max_num_seqs 1 \
  --enable-expert-parallel

Full Logs

Click to expand full vLLM service log and nvidia-smi output 
(APIServer pid=77870) INFO 10-23 17:11:49 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0%
(APIServer pid=77870) INFO:     127.0.0.1:59562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77870) INFO 10-23 17:11:59 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=77870) INFO 10-23 17:12:09 [loggers.py:127] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=77870) INFO:     127.0.0.1:59562 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:59562 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:59562 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60466 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60466 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60466 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:32928 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:32928 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:32928 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
[rank4]:[E1023 17:16:46.022239569 ProcessGroupNCCL.cpp:2068] [PG ID 2 PG GUID 3 Rank 4] Process group watchdog thread terminated with exception: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x7f6c240d9eb0 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x111c7 (0x7f6c2416c1c7 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x50 (0x7f6bc76c4640 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x68 (0x7f6bc76d3e28 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x978 (0x7f6bc76d6f48 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0xd2 (0x7f6bc76d8ec2 in /home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdbbf4 (0x7f6bab05abf4 in /home/lt_08321/miniconda3/envs/qwenvl/bin/../lib/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f6c24e63ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7f6c24ef4bf4 in /lib/x86_64-linux-gnu/libc.so.6)

(APIServer pid=77870) INFO:     127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=77870) INFO:     127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(EngineCore_DP0 pid=78020) INFO 10-23 17:17:39 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
(APIServer pid=77870) INFO:     127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:47 [multiproc_executor.py:154] Worker proc VllmWorker-4 died unexpectedly, shutting down executor.
(Worker_TP0_EP0 pid=78171) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP1_EP1 pid=78173) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP2_EP2 pid=78180) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP3_EP3 pid=78181) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP5_EP5 pid=78183) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP6_EP6 pid=78184) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP7_EP7 pid=78185) INFO 10-23 17:17:47 [multiproc_executor.py:558] Parent process exited, terminating worker
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0) with config: model='checkpoints/Qwen/Qwen3-VL-235B-A22B-Instruct', speculative_config=None, tokenizer='checkpoints/Qwen/Qwen3-VL-235B-A22B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=124000, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwenvl, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":4,"local_cache_dir":null},
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0170e13ac331471da7dd9a65d57eb9ae'], resumed_from_preemption=[false], new_token_ids=[], new_block_ids=[[[7927, 7928, 7929, 7930, 7931, 7932, 7933, 7934, 7935, 7936, 7937, 7938, 7939, 7940, 7941, 7942, 7943, 7944, 7945, 7946, 7947, 7948, 7949, 7950, 7951, 7952, 7953, 7954, 7955, 7956, 7957, 7958, 7959, 7960, 7961, 7962, 7963, 7964, 7965, 7966, 7967, 7968, 7969, 7970, 7971, 7972, 7973, 7974, 7975, 7976, 7977, 7978, 7979, 7980, 7981, 7982, 7983, 7984, 7985, 7986, 7987, 7988, 7989, 7990, 7991, 7992, 7993, 7994, 7995, 7996, 7997, 7998, 7999, 8000, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8008, 8009, 8010, 8011, 8012, 8013, 8014, 8015, 8016, 8017, 8018, 8019, 8020, 8021, 8022, 8023, 8024, 8025, 8026, 8027, 8028, 8029, 8030, 8031, 8032, 8033, 8034, 8035, 8036, 8037, 8038, 8039, 8040, 8041, 8042, 8043, 8044, 8045, 8046, 8047, 8048, 8049, 8050, 8051, 8052, 8053, 8054]]], num_computed_tokens=[8192]), num_scheduled_tokens={chatcmpl-0170e13ac331471da7dd9a65d57eb9ae: 2048}, total_num_scheduled_tokens=2048, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[640], finished_req_ids=[], free_encoder_mm_hashes=[], structured_output_request_ids={chatcmpl-0170e13ac331471da7dd9a65d57eb9ae: 0}, grammar_bitmask=array([[       0,        0, 67108864, ...,        0,        0,        0]],
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [dump_input.py:76]       shape=(1, 4748), dtype=int32), kv_connector_metadata=null)
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.03260980332212371, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, kv_connector_stats=None, num_corrupted_reqs=0)
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)] Traceback (most recent call last):
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 728, in run_busy_loop
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     self._process_engine_step()
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 284, in step
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 270, in execute_model_with_error_logging
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     raise err
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 261, in execute_model_with_error_logging
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     return model_fn(scheduler_output)
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 181, in execute_model
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     (output, ) = self.collective_rpc(
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]                  ^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 264, in collective_rpc
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     result = get_response(w, dequeue_timeout,
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 244, in get_response
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     status, result = w.worker_response_mq.dequeue(
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 511, in dequeue
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     with self.acquire_read(timeout, cancel, indefinite) as buf:
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/contextlib.py", line 137, in __enter__
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     return next(self.gen)
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]            ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 455, in acquire_read
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)]     raise RuntimeError("cancelled")
(EngineCore_DP0 pid=78020) ERROR 10-23 17:17:52 [[core.py:710](http://core.py:710/)] RuntimeError: cancelled
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480] AsyncLLM output_handler failed.
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480] Traceback (most recent call last):
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 439, in output_handler
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480]     outputs = await engine_core.get_output_async()
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=77870) ERROR 1SAPIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480]   File "/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 846, in get_output_async
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480]     raise self._format_exception(outputs) from None
(APIServer pid=77870) ERROR 10-23 17:17:52 [async_llm.py:480] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=77870) INFO:     Shutting down
(APIServer pid=77870) INFO:     Waiting for application shutdown.
(APIServer pid=77870) INFO:     Application shutdown complete.
(APIServer pid=77870) INFO:     Finished server process [77870]
nanobind: leaked 2 instances!
 - leaked instance 0x7f335d9ed2a8 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7f335d9ed338 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
nanobind: leaked 2 types!
 - leaked type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked type "xgrammar.xgrammar_bindings.CompiledGrammar"
nanobind: leaked 16 functions!
 - leaked function "reset"
 - leaked function ""
 - leaked function ""
 - leaked function "__init__"
 - leaked function "is_terminated"
 - leaked function "accept_token"
 - leaked function "_debug_print_internal_state"
 - leaked function "deserialize_json"
 - leaked function ""
 - leaked function ""
 - leaked function ""
 - leaked function "serialize_json"
 - leaked function "find_jump_forward_string"
 - leaked function "fill_next_token_bitmask"
 - leaked function "rollback"
 - leaked function "accept_string"
nanobind: this is likely caused by a reference counting issue in the binding code.
^C/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 8 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/lt_08321/miniconda3/envs/qwenvl/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 9 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

---
**`nvidia-smi` output (taken *after* the crash, while no processes were running)**
---`

Thu Oct 23 17:19:04 2025

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 12.1     |

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|                               |                      |               MIG M. |

|===============================+======================+======================|

|   0  NVIDIA A100-SXM...  Off  | 00000000:13:00.0 Off |                    0 |

| N/A   32C    P0    66W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   1  NVIDIA A100-SXM...  Off  | 00000000:19:00.0 Off |                    0 |

| N/A   29C    P0    67W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   2  NVIDIA A100-SXM...  Off  | 00000000:48:00.0 Off |                    0 |

| N/A   29C    P0    60W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   3  NVIDIA A100-SXM...  Off  | 00000000:4D:00.0 Off |                    0 |

| N/A   32C    P0    67W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   4  NVIDIA A100-SXM...  Off  | 00000000:89:00.0 Off |                    0 |

| N/A   31C    P0   ERR! / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   5  NVIDIA A100-SXM...  Off  | 00000000:8E:00.0 Off |                    0 |

| N/A   30C    P0   ERR! / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   6  NVIDIA A100-SXM...  Off  | 00000000:AD:00.0 Off |                    0 |

| N/A   29C    P0    63W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

|   7  NVIDIA A100-SXM...  Off  | 00000000:B3:00.0 Off |                    0 |

| N/A   32C    P0    66W / 400W |      0MiB / 81251MiB |      0%      Default |

|                               |                      |             Disabled |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes:                                                                  |

|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |

|        ID   ID                                                   Usage      |

|=============================================================================|

|  No running processes found                                                 |

+-----------------------------------------------------------------------------+

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingnvidia

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions