[Bug]: illegal memory access when there are multiple concurrent request

[seabnavin19]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.1.0
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.1+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-4.18.0-553.56.1.el8_10.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration :
GPU 0: NVIDIA A16
GPU 1: NVIDIA A16
GPU 2: NVIDIA A16
GPU 3: NVIDIA A16
GPU 4: NVIDIA A16
GPU 5: NVIDIA A16
GPU 6: NVIDIA A16
GPU 7: NVIDIA A16
GPU 8: NVIDIA A16
GPU 9: NVIDIA A16
GPU 10: NVIDIA A16
GPU 11: NVIDIA A16
GPU 12: NVIDIA A16
GPU 13: NVIDIA A16
GPU 14: NVIDIA A16
GPU 15: NVIDIA A16

Nvidia driver version        : 575.57.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        45 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               32
On-line CPU(s) list:                  0-31
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Gold 6442Y
CPU family:                           6
Model:                                79
Thread(s) per core:                   1
Core(s) per socket:                   1
Socket(s):                            32
Stepping:                             0
BogoMIPS:                             5199.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt arat md_clear flush_l1d arch_capabilities
Hypervisor vendor:                    VMware
Virtualization type:                  full
L1d cache:                            1.5 MiB (32 instances)
L1i cache:                            1 MiB (32 instances)
L2 cache:                             64 MiB (32 instances)
L3 cache:                             1.9 GiB (32 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-31
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                   Mitigation; PTE Inversion
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==27.0.1
[pip3] torch==2.7.1+cu128
[pip3] torchaudio==2.7.1+cu128
[pip3] torchvision==0.22.1+cu128
[pip3] transformers==4.55.2
[pip3] triton==3.3.1
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.10.1.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    GPU8    GPU9    GPU10   GPU11   GPU12   GPU13   GPU14   GPU15   CPU Affinity    NUMA Affinity        GPU NUMA ID
GPU0     X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU1    PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU2    PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU3    PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU4    PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU5    PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU6    PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU7    PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU8    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU9    PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU10   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     0-31    0           N/A
GPU11   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     0-31    0           N/A
GPU12   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     0-31    0           N/A
GPU13   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     0-31    0           N/A
GPU14   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     0-31    0           N/A
GPU15   PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      0-31    0           N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
CUDA_LAUNCH_BLOCKING=1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
VLLM_LOGGING_LEVEL=ERROR
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I have deploy the vLLM with Qwen3 model in docker with below files

Dockerfile

FROM vllm/vllm-openai:latest
COPY . .
RUN pip install -r requirements.txt

docker-compose.yaml

services:
  vllm-openai:
   # image: vllm/vllm-openai:v0.9.2
    build: .
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    command: >
      --model Qwen/Qwen3-30B-A3B-Instruct-2507
      --host 0.0.0.0
      --port 8000
      --swap-space 2
      --tensor-parallel-size 8
      --pipeline-parallel-size 2
      --gpu-memory-utilization 0.85
      --enable-auto-tool-choice
      --tool-call-parser xlam
      --middleware middlewares.tracing.phoenix_middleware

    environment:
      - HUGGING_FACE_HUB_TOKEN= ${HUGGING_FACE_HUB_TOKEN}
      - VLLM_LOGGING_LEVEL=ERROR
      - PHOENIX_API_KEY=${PHOENIX_API_KEY}
      - CUDA_LAUNCH_BLOCKING=1
    runtime: nvidia
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - ./middlewares:/vllm-workspace/middlewares
    gpus: all
    ipc: host
    shm_size: 64g

The deployment was functioning correctly under normal conditions, but in production, when handling multiple concurrent requests from different applications, it consistently triggers an illegal memory access error. The exact cause is unclear, but it appears to be related to a memory leak or improper memory management of GPU.

{"log":"  what():  [PG ID 2 PG GUID 5 Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered\n","stream":"stderr","time":"2025-08-28T08:07:28.399792365Z"}
{"log":"Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n","stream":"stderr","time":"2025-08-28T08:07:28.399800066Z"}
{"log":"\n","stream":"stderr","time":"2025-08-28T08:07:28.39980311Z"}
{"log":"Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):\n","stream":"stderr","time":"2025-08-28T08:07:28.39980593Z"}
{"log":"frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e) + 0x98 (0x7f9d934005e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399808921Z"}
{"log":"frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e const\u0026) + 0xe0 (0x7f9d933954a2 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399812696Z"}
{"log":"frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7f9dfe5a5422 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399816073Z"}
{"log":"frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f9d9416d5a6 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399819069Z"}
{"log":"frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7f9d9417d840 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399822131Z"}
{"log":"frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7f9d9417f3d2 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399825035Z"}
{"log":"frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f9d94180fdd in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399828061Z"}
{"log":"frame #7: \u003cunknown function\u003e + 0xdc253 (0x7f9d843b3253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399830975Z"}
{"log":"frame #8: \u003cunknown function\u003e + 0x94ac3 (0x7f9dff1bbac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399834018Z"}
{"log":"frame #9: clone + 0x44 (0x7f9dff24ca04 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.39984039Z"}
{"log":"\n","stream":"stderr","time":"2025-08-28T08:07:28.399843375Z"}
{"log":"Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):\n","stream":"stderr","time":"2025-08-28T08:07:28.399846241Z"}
{"log":"frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e) + 0x98 (0x7f9d934005e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399849501Z"}
{"log":"frame #1: \u003cunknown function\u003e + 0xcc7b9e (0x7f9d9414fb9e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399852764Z"}
{"log":"frame #2: \u003cunknown function\u003e + 0x9165ed (0x7f9d93d9e5ed in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399855693Z"}
{"log":"frame #3: \u003cunknown function\u003e + 0xdc253 (0x7f9d843b3253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399861995Z"}
{"log":"frame #4: \u003cunknown function\u003e + 0x94ac3 (0x7f9dff1bbac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399865058Z"}
{"log":"frame #5: clone + 0x44 (0x7f9dff24ca04 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399868161Z"}
{"log":"\n","stream":"stderr","time":"2025-08-28T08:07:28.39987103Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] WorkerProc hit an exception.\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153349288Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] Traceback (most recent call last):\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153398453Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 592, in worker_busy_loop\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153407068Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     output = func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153412434Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15341686Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 116, in decorate_context\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15342108Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153425725Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15342972Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 369, in execute_model\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153433826Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     get_pp_group().send_tensor_dict(output.tensors,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153438212Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 700, in send_tensor_dict\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153442303Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     self.send_object(metadata_list, dst=dst)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153455564Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 530, in send_object\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153461165Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     torch.distributed.send(size_tensor,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153465552Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153469772Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153474117Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153478112Z"}
{"log":"frame #2: \u003cunknown function\u003e + 0x9165ed (0x7f9d93d9e5ed in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:28.399855693Z"}
{"log":"frame #3: \u003cunknown function\u003e + 0xdc253 (0x7f9d843b3253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399861995Z"}
{"log":"frame #4: \u003cunknown function\u003e + 0x94ac3 (0x7f9dff1bbac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399865058Z"}
{"log":"frame #5: clone + 0x44 (0x7f9dff24ca04 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:28.399868161Z"}
{"log":"\n","stream":"stderr","time":"2025-08-28T08:07:28.39987103Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] WorkerProc hit an exception.\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153349288Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] Traceback (most recent call last):\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153398453Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 592, in worker_busy_loop\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153407068Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     output = func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153412434Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15341686Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 116, in decorate_context\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15342108Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153425725Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15342972Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 369, in execute_model\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153433826Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     get_pp_group().send_tensor_dict(output.tensors,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153438212Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 700, in send_tensor_dict\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153442303Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     self.send_object(metadata_list, dst=dst)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153455564Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 530, in send_object\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153461165Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     torch.distributed.send(size_tensor,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153465552Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153469772Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153474117Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153478112Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py\", line 2436, in send\r\n","stream":"stdout","time":"2025-08-28T08:07:33.15348207Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     work.wait()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153486357Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.20.0.2]:25746\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153490357Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] Traceback (most recent call last):\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153494633Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 592, in worker_busy_loop\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153498652Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     output = func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153501827Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153504749Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 116, in decorate_context\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153507737Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153511022Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153514197Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 369, in execute_model\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153517124Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     get_pp_group().send_tensor_dict(output.tensors,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153524233Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 700, in send_tensor_dict\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153527318Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     self.send_object(metadata_list, dst=dst)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153530676Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py\", line 530, in send_object\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153533589Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     torch.distributed.send(size_tensor,\r\n","stream":"stdout","time":"2025-08-28T08:07:33.1535367Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153539655Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     return func(*args, **kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153542699Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153545562Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]   File \"/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py\", line 2436, in send\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153548429Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597]     work.wait()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153551487Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.20.0.2]:25746\r\n","stream":"stdout","time":"2025-08-28T08:07:33.153554631Z"}
{"log":"\u001b[1;36m(VllmWorker PP0_TP0 pid=212)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:597] \n","stream":"stdout","time":"2025-08-28T08:07:33.153557761Z"}
"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [multiproc_executor.py:146] Worker proc VllmWorker-8 died unexpectedly, shutting down executor.\n","stream":"stdout","time":"2025-08-28T08:07:33.195502497Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.1.1) with config: model='Qwen/Qwen3-30B-A3B-Instruct-2507', speculative_config=None, tokenizer='Qwen/Qwen3-30B-A3B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-30B-A3B-Instruct-2507, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={\"level\":3,\"debug_dump_path\":\"\",\"cache_dir\":\"\",\"backend\":\"\",\"custom_ops\":[],\"splitting_ops\":[\"vllm.unified_attention\",\"vllm.unified_attention_with_output\",\"vllm.mamba_mixer2\"],\"use_inductor\":true,\"compile_sizes\":[],\"inductor_compile_config\":{\"enable_auto_functionalized_v2\":false},\"inductor_passes\":{},\"cudagraph_mode\":1,\"use_cudagraph\":true,\"cudagraph_num_of_warmups\":1,\"cudagraph_capture_sizes\":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],\"cudagraph_copy_inputs\":false,\"full_cuda_graph\":false,\"pass_config\":{},\"max_capture_size\":512,\"local_cache_dir\":null}, \n","stream":"stdout","time":"2025-08-28T08:07:33.197960071Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-45148ecf8f8e4f94b8fad390770ad7aa,prompt_token_ids_len=3177,mm_kwargs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=5120, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json={'type': 'array', 'minItems': 1, 'items': {'type': 'object', 'anyOf': [{'properties': {'name': {'type': 'string', 'enum': ['TextToSQL']}, 'parameters': {'type': 'object', 'properties': {'question': {'type': 'string', 'description': 'Analytical question in natural language'}, 'user_id': {'type': 'string', 'description': 'User identifier for session tracking'}}, 'required': ['question', 'user_id']}}, 'required': ['name', 'parameters']}]}}, regex=None, choice=None, grammar=None, json_object=None, backend='xgrammar', backend_was_auto=True, disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, whitespace_pattern=None, structural_tag=None), extra_args=None),block_ids=([1419, 1420, 1421, 1422, 1423, 1424, 1425, 1426, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 1482, 1483, 1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 1518, 1519, 1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 1530, 1531, 1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 1542, 1543, 1544, 1545, 1546],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-d49596783d3e44ec81d57dc61333f306'], resumed_from_preemption=[false], new_token_ids=[[374]], new_block_ids=[[[]]], num_computed_tokens=[5124]), num_scheduled_tokens={chatcmpl-45148ecf8f8e4f94b8fad390770ad7aa: 2047, chatcmpl-d49596783d3e44ec81d57dc61333f306: 1}, total_num_scheduled_tokens=2048, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={chatcmpl-d49596783d3e44ec81d57dc61333f306: 0, chatcmpl-45148ecf8f8e4f94b8fad390770ad7aa: 1}, grammar_bitmask=array([[       0, 67108864,        0, ...,        0,        0,        0],\r\n","stream":"stdout","time":"2025-08-28T08:07:33.198629732Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [dump_input.py:76]        [       0, 67108864,        0, ...,        0,        0,        0]],\r\n","stream":"stdout","time":"2025-08-28T08:07:33.198672289Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [dump_input.py:76]       shape=(2, 4748), dtype=int32), kv_connector_metadata=null)\n","stream":"stdout","time":"2025-08-28T08:07:33.198676001Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.01181941923774954, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=3177, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)\n","stream":"stdout","time":"2025-08-28T08:07:33.199082171Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702] EngineCore encountered a fatal error.\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207579063Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702] Traceback (most recent call last):\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207601244Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 693, in run_engine_core\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207605253Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     engine_core.run_busy_loop()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207608864Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 720, in run_busy_loop\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207611238Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     self._process_engine_step()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207613809Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 745, in _process_engine_step\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207616181Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     outputs, model_executed = self.step_fn()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207618735Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]                               ^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207621145Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 339, in step_with_batch_queue\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207623438Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     model_output = self.execute_model_with_error_logging(\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207626171Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207628581Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 274, in execute_model_with_error_logging\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207630921Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     raise err\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207633374Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 265, in execute_model_with_error_logging\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207641757Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     return model_fn(scheduler_output)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207644357Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]            ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207646938Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py\", line 340, in \u003clambda\u003e\r\n","stream":"stdout","time":"2025-08-28T08:07:33.20764922Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     lambda _: future.result(), scheduler_output)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207652153Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]               ^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207654969Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 456, in result\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207657246Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     return self.__get_result()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207659676Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]            ^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207661899Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207664122Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     raise self._exception\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207666489Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/lib/python3.12/concurrent/futures/thread.py\", line 59, in run\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207668726Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     result = self.fn(*self.args, **self.kwargs)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207671509Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207673792Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 226, in get_response\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207676028Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     status, result = w.worker_response_mq.dequeue(\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207678516Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207680974Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py\", line 511, in dequeue\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207683466Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     with self.acquire_read(timeout, cancel) as buf:\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207685953Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207693099Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/lib/python3.12/contextlib.py\", line 137, in __enter__\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207695488Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     return next(self.gen)\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207697875Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]            ^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207700132Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]   File \"/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py\", line 468, in acquire_read\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207702451Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702]     raise RuntimeError(\"cancelled\")\r\n","stream":"stdout","time":"2025-08-28T08:07:33.207705099Z"}
{"log":"\u001b[1;36m(EngineCore_0 pid=142)\u001b[0;0m ERROR 08-28 01:07:33 [core.py:702] RuntimeError: cancelled\n","stream":"stdout","time":"2025-08-28T08:07:33.207707429Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430] AsyncLLM output_handler failed.\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213294943Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430] Traceback (most recent call last):\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213338072Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py\", line 389, in output_handler\r\n","stream":"stdout","time":"2025-08-28T08:07:33.2133437Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430]     outputs = await engine_core.get_output_async()\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213347866Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213350996Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430]   File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py\", line 843, in get_output_async\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213353892Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430]     raise self._format_exception(outputs) from None\r\n","stream":"stdout","time":"2025-08-28T08:07:33.213357146Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m ERROR 08-28 01:07:33 [async_llm.py:430] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.\n","stream":"stdout","time":"2025-08-28T08:07:33.213360092Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     10.4.4.78:37160 - \"POST /v1/chat/completions HTTP/1.1\" 500 Internal Server Error\n","stream":"stdout","time":"2025-08-28T08:07:33.219084807Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     10.4.4.78:37166 - \"POST /v1/chat/completions HTTP/1.1\" 500 Internal Server Error\n","stream":"stdout","time":"2025-08-28T08:07:33.224559296Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     Shutting down\n","stream":"stderr","time":"2025-08-28T08:07:33.289775473Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     Waiting for application shutdown.\n","stream":"stderr","time":"2025-08-28T08:07:33.390138122Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     Application shutdown complete.\n","stream":"stderr","time":"2025-08-28T08:07:33.390361019Z"}
{"log":"\u001b[1;36m(APIServer pid=1)\u001b[0;0m INFO:     Finished server process [1]\n","stream":"stderr","time":"2025-08-28T08:07:33.390383871Z"}
{"log":"nanobind: leaked 6 instances!\n","stream":"stderr","time":"2025-08-28T08:07:34.356967992Z"}
{"log":" - leaked instance 0x7f76a28bb378 of type \"xgrammar.xgrammar_bindings.GrammarMatcher\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357072259Z"}
{"log":" - leaked instance 0x7f76a28bb318 of type \"xgrammar.xgrammar_bindings.GrammarMatcher\"\n","stream":"stderr","time":"2025-08-28T08:07:34.35708971Z"}
{"log":" - leaked instance 0x7f76a2924858 of type \"xgrammar.xgrammar_bindings.CompiledGrammar\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357100354Z"}
{"log":" - leaked instance 0x7f76a278a688 of type \"xgrammar.xgrammar_bindings.GrammarMatcher\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357110649Z"}
{"log":" - leaked instance 0x7f76a2926ef8 of type \"xgrammar.xgrammar_bindings.CompiledGrammar\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357122026Z"}
{"log":" - leaked instance 0x7f76a28bb138 of type \"xgrammar.xgrammar_bindings.CompiledGrammar\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357131278Z"}
{"log":"nanobind: leaked 2 types!\n","stream":"stderr","time":"2025-08-28T08:07:34.357140353Z"}
{"log":" - leaked type \"xgrammar.xgrammar_bindings.GrammarMatcher\"\n","stream":"stderr","time":"2025-08-28T08:07:34.35714905Z"}
{"log":" - leaked type \"xgrammar.xgrammar_bindings.CompiledGrammar\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357158352Z"}
{"log":"nanobind: leaked 16 functions!\n","stream":"stderr","time":"2025-08-28T08:07:34.35716745Z"}
{"log":" - leaked function \"__init__\"\n","stream":"stderr","time":"2025-08-28T08:07:34.35717613Z"}
{"log":" - leaked function \"\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357185193Z"}
{"log":" - leaked function \"fill_next_token_bitmask\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357194417Z"}
{"log":" - leaked function \"serialize_json\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357203997Z"}
{"log":" - leaked function \"find_jump_forward_string\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357213148Z"}
{"log":" - leaked function \"rollback\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357222212Z"}
{"log":" - leaked function \"\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357231007Z"}
{"log":" - leaked function \"\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357239564Z"}
{"log":" - leaked function \"reset\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357248372Z"}
{"log":" - leaked function \"\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357286136Z"}
{"log":" - leaked function \"\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357295331Z"}
{"log":" - leaked function \"_debug_print_internal_state\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357303891Z"}
{"log":" - leaked function \"deserialize_json\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357312888Z"}
{"log":" - leaked function \"accept_token\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357321735Z"}
{"log":" - leaked function \"accept_string\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357330475Z"}
{"log":" - leaked function \"is_terminated\"\n","stream":"stderr","time":"2025-08-28T08:07:34.357339207Z"}
{"log":"nanobind: this is likely caused by a reference counting issue in the binding code.\n","stream":"stderr","time":"2025-08-28T08:07:34.357348023Z"}
{"log":"[rank9]:[W828 01:07:36.293269650 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=170, addr=[localhost]:52084, remote=[localhost]:58815): failed to recv, got 0 bytes\n","stream":"stderr","time":"2025-08-28T08:07:36.019040067Z"}
{"log":"Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):\n","stream":"stderr","time":"2025-08-28T08:07:36.01908949Z"}
{"log":"frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e) + 0x98 (0x7fb914d785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.019094393Z"}
{"log":"frame #1: \u003cunknown function\u003e + 0x5ba8bfe (0x7fb8f92dcbfe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.019104829Z"}
{"log":"frame #2: \u003cunknown function\u003e + 0x5baaf40 (0x7fb8f92def40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.01910802Z"}
{"log":"frame #3: \u003cunknown function\u003e + 0x5bab84a (0x7fb8f92df84a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.019111256Z"}
{"log":"frame #4: c10d::TCPStore::check(std::vector\u003cstd::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e, std::allocator\u003cstd::__cxx11::basic_string\u003cchar, std::char_traits\u003cchar\u003e, std::allocator\u003cchar\u003e \u003e \u003e \u003e const\u0026) + 0x2a9 (0x7fb8f92d92a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.019114737Z"}
{"log":"frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7fb8aa97bad9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\n","stream":"stderr","time":"2025-08-28T08:07:36.019118858Z"}
{"log":"frame #6: \u003cunknown function\u003e + 0xdc253 (0x7fb89adb3253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:36.019122101Z"}
{"log":"frame #7: \u003cunknown function\u003e + 0x94ac3 (0x7fb915a59ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:36.019124995Z"}
{"log":"frame #8: clone + 0x44 (0x7fb915aeaa04 in /usr/lib/x86_64-linux-gnu/libc.so.6)\n","stream":"stderr","time":"2025-08-28T08:07:36.019127894Z"}
{"log":"\n","stream":"stderr","time":"2025-08-28T08:07:36.019130676Z"}
{"log":"[rank9]:[W828 01:07:36.310492869 ProcessGroupNCCL.cpp:1662] [PG ID 0 PG GUID 0 Rank 9] Failed to check the \"should dump\" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes\n","stream":"stderr","time":"2025-08-28T08:07:36.019133431Z"}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[njhill]

@seabnavin19 I am working on a PR which might help with this: #23770

[seabnavin19]

I having this error only when I use force tool calling , I'm not sure if this PR related to my issue

[gdagil]

+1

My setup - 4xH100

ENV:

USE_FASTSAFETENSOR: "true"
VLLM_FLASH_ATTN_VERSION: '3'
TORCH_USE_CUDA_DSA: "1"
TF_CPP_MIN_LOG_LEVEL: "3"
NCCL_CUMEM_ENABLE: "true"
CUDA_LAUNCH_BLOCKING: "1"

COMMAND:

vllm serve \
  --host 0.0.0.0 \
  --port 8080 \
    /models/deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
    --served-model-name deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.9 \
    --tensor-parallel-size 4 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-prompt-tokens-details \
    --enable-force-include-usage \
    --reasoning-parser deepseek_r1

I have same error in SO tasks in v0.10.0 and v0.10.1.1 version

�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] WorkerProc hit an exception.
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=0 pid=222)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] WorkerProc hit an exception.
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=3 pid=225)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] WorkerProc hit an exception.
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Traceback (most recent call last):
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 541, in worker_busy_loop
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     output = self.model_runner.execute_model(scheduler_output,
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     return func(*args, **kwargs)
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1582, in execute_model
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]     valid_sampled_token_ids = sampled_token_ids.tolist()
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] RuntimeError: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
�[1;36m(VllmWorker rank=2 pid=224)�[0;0m ERROR 08-29 01:18:42 [multiproc_executor.py:546]
ERROR 08-29 01:18:42 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.1.dev1+gbcc0a3cbe) with config: model='/models/deepseek-ai/DeepSeek-R1-Distill-Llama-70B', speculative_config=None, tokenizer='/models/deepseek-ai/DeepSeek-R1-Distill-Llama-70B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=deepseek-ai/DeepSeek-R1-Distill-Llama-70B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null},
ERROR 08-29 01:18:42 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-69d1af6d-acca-439d-96f6-72d58035ad4d', 'chatcmpl-a19674fa-1454-4f8a-9dca-8e32e240ca03', 'chatcmpl-d67abede-b577-4170-93bf-4955a6dc3f1f', 'chatcmpl-7858892f-ed8a-4281-b95d-3450e7bda682', 'chatcmpl-8b335cd3-c1ab-428c-bc2d-8eafb9a8d616', 'chatcmpl-eb635c77-cea2-46a4-bcd1-e99ca464ad6e'], resumed_from_preemption=[false, false, false, false, false, false], new_token_ids=[], new_block_ids=[[[]], [[]], [[]], [[]], [[]], [[]]], num_computed_tokens=[3231, 2469, 2316, 3109, 3858, 7]), num_scheduled_tokens={chatcmpl-8b335cd3-c1ab-428c-bc2d-8eafb9a8d616: 1, chatcmpl-a19674fa-1454-4f8a-9dca-8e32e240ca03: 1, chatcmpl-69d1af6d-acca-439d-96f6-72d58035ad4d: 1, chatcmpl-7858892f-ed8a-4281-b95d-3450e7bda682: 1, chatcmpl-d67abede-b577-4170-93bf-4955a6dc3f1f: 1, chatcmpl-eb635c77-cea2-46a4-bcd1-e99ca464ad6e: 1}, total_num_scheduled_tokens=6, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={chatcmpl-7858892f-ed8a-4281-b95d-3450e7bda682: 3, chatcmpl-a19674fa-1454-4f8a-9dca-8e32e240ca03: 1, chatcmpl-69d1af6d-acca-439d-96f6-72d58035ad4d: 0, chatcmpl-8b335cd3-c1ab-428c-bc2d-8eafb9a8d616: 4, chatcmpl-d67abede-b577-4170-93bf-4955a6dc3f1f: 2}, grammar_bitmask=array([[      -1,       -1,       -1, ...,       -1,       -1,       -1],
ERROR 08-29 01:18:42 [dump_input.py:76]        [       0,        0, 67108864, ...,        0,        0,        0],
ERROR 08-29 01:18:42 [dump_input.py:76]        [      -1,       -1,       -1, ...,       -1,       -1,       -1],
ERROR 08-29 01:18:42 [dump_input.py:76]        [      -1,       -1,       -1, ...,       -1,       -1,       -1],
ERROR 08-29 01:18:42 [dump_input.py:76]        [      -1,       -1,       -1, ...,       -1,       -1,       -1]],
ERROR 08-29 01:18:42 [dump_input.py:76]       shape=(5, 4008), dtype=int32), kv_connector_metadata=null)
ERROR 08-29 01:18:42 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=6, num_waiting_reqs=0, kv_cache_usage=0.01945942085907093, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
ERROR 08-29 01:18:42 [core.py:634] EngineCore encountered a fatal error.
ERROR 08-29 01:18:42 [core.py:634] Traceback (most recent call last):
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 08-29 01:18:42 [core.py:634]     engine_core.run_busy_loop()
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 08-29 01:18:42 [core.py:634]     self._process_engine_step()
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 08-29 01:18:42 [core.py:634]     outputs, model_executed = self.step_fn()
ERROR 08-29 01:18:42 [core.py:634]                               ^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 267, in step
ERROR 08-29 01:18:42 [core.py:634]     model_output = self.execute_model_with_error_logging(
ERROR 08-29 01:18:42 [core.py:634]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging
ERROR 08-29 01:18:42 [core.py:634]     raise err
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging
ERROR 08-29 01:18:42 [core.py:634]     return model_fn(scheduler_output)
ERROR 08-29 01:18:42 [core.py:634]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 167, in execute_model
ERROR 08-29 01:18:42 [core.py:634]     (output, ) = self.collective_rpc(
ERROR 08-29 01:18:42 [core.py:634]                  ^^^^^^^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 237, in collective_rpc
ERROR 08-29 01:18:42 [core.py:634]     result = get_response(w, dequeue_timeout)
ERROR 08-29 01:18:42 [core.py:634]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 224, in get_response
ERROR 08-29 01:18:42 [core.py:634]     raise RuntimeError(
ERROR 08-29 01:18:42 [core.py:634] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
ERROR 08-29 01:18:42 [core.py:634] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 08-29 01:18:42 [core.py:634] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 08-29 01:18:42 [core.py:634] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 08-29 01:18:42 [core.py:634] ', please check the stack trace above for the root cause
ERROR 08-29 01:18:42 [async_llm.py:416] AsyncLLM output_handler failed.
ERROR 08-29 01:18:42 [async_llm.py:416] Traceback (most recent call last):
ERROR 08-29 01:18:42 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler
ERROR 08-29 01:18:42 [async_llm.py:416]     outputs = await engine_core.get_output_async()
ERROR 08-29 01:18:42 [async_llm.py:416]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-29 01:18:42 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async
ERROR 08-29 01:18:42 [async_llm.py:416]     raise self._format_exception(outputs) from None
ERROR 08-29 01:18:42 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-69d1af6d-acca-439d-96f6-72d58035ad4d failed (engine dead).
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-a19674fa-1454-4f8a-9dca-8e32e240ca03 failed (engine dead).
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-d67abede-b577-4170-93bf-4955a6dc3f1f failed (engine dead).
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-7858892f-ed8a-4281-b95d-3450e7bda682 failed (engine dead).
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-8b335cd3-c1ab-428c-bc2d-8eafb9a8d616 failed (engine dead).
INFO 08-29 01:18:42 [async_llm.py:342] Request chatcmpl-eb635c77-cea2-46a4-bcd1-e99ca464ad6e failed (engine dead).
INFO:     10.233.74.227:52746 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     10.233.74.227:38222 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     10.233.74.227:51970 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     10.233.74.227:38224 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     10.233.74.227:38226 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     10.233.74.227:34772 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
[rank2]:[W829 01:18:43.561201163 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=85, addr=[::ffff:127.0.0.1]:39842, remote=[::ffff:127.0.0.1]:42693): Connection reset by peer
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f53e32005e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8bfe (0x7f54328dcbfe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baafcf (0x7f54328defcf in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab84a (0x7f54328df84a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7f54328d92a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7f53e3f7bad9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f53d41b3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f544ef73ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7f544f004a04 in /lib/x86_64-linux-gnu/libc.so.6)

[rank2]:[W829 01:18:43.570756281 ProcessGroupNCCL.cpp:1662] [PG ID 0 PG GUID 0 Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer
[rank3]:[W829 01:18:43.561201003 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=85, addr=[::ffff:127.0.0.1]:39826, remote=[::ffff:127.0.0.1]:42693): Connection reset by peer
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7fd5d53785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8bfe (0x7fd5b98dcbfe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baafcf (0x7fd5b98defcf in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab84a (0x7fd5b98df84a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7fd5b98d92a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7fd56af7bad9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7fd55b3b3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7fd5d6058ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7fd5d60e9a04 in /lib/x86_64-linux-gnu/libc.so.6)

[rank3]:[W829 01:18:43.576370238 ProcessGroupNCCL.cpp:1662] [PG ID 0 PG GUID 0 Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer
[rank1]:[W829 01:18:43.561203195 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=85, addr=[::ffff:127.0.0.1]:39818, remote=[::ffff:127.0.0.1]:42693): Connection reset by peer
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f91afc005e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8bfe (0x7f91ff2dcbfe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baafcf (0x7f91ff2defcf in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab84a (0x7f91ff2df84a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7f91ff2d92a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7f91b097bad9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f91a0bb3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f921b912ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7f921b9a3a04 in /lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[W829 01:18:43.577939385 ProcessGroupNCCL.cpp:1662] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer
nanobind: leaked 10 instances!
 - leaked instance 0x7fb1ee9077f8 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked instance 0x7fb1eec462f8 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7fb1ef614738 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7fb1ef54ceb8 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked instance 0x7fb1ef190168 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7fb1eff80e88 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7fb1ef923a08 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked instance 0x7fb1ee906418 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked instance 0x7fb1ef9200d8 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x7fb1ef922e98 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
nanobind: leaked 2 types!
 - leaked type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked type "xgrammar.xgrammar_bindings.CompiledGrammar"
nanobind: leaked 16 functions!
 - leaked function ""
 - leaked function "fill_next_token_bitmask"
 - leaked function "serialize_json"
 - leaked function "reset"
 - leaked function ""
 - leaked function "deserialize_json"
 - leaked function "accept_token"
 - leaked function ""
 - leaked function "find_jump_forward_string"
 - leaked function "rollback"
 - leaked function ""
 - leaked function "is_terminated"
 - leaked function "accept_string"
 - leaked function ""
 - leaked function "_debug_print_internal_state"
 - leaked function "__init__"
nanobind: this is likely caused by a reference counting issue in the binding code.
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[njhill]

@seabnavin19 I am working on a PR which might help with this: #23770

This PR is now merged but actually most likely won't help. I thought it could be to do with use of pipeline parallel but looks like this is happening in non-PP cases too.

[ItzAmirreza]

I'm having the same issue.

--host 0.0.0.0 --port 8000 --model Kwaipilot/KAT-V1-40B --gpu-memory-utilization 0.95 --api-key [redacted] --max-model-len 10000 --served-model-name fmax --enable-auto-tool-choice --tool-call-parser hermes --guided-decoding-backend auto --tensor-parallel-size 2

error:

2025-09-02T08:04:02.874454226-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] WorkerProc hit an exception.
2025-09-02T08:04:02.874480216-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Traceback (most recent call last):
2025-09-02T08:04:02.874484356-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 591, in worker_busy_loop
2025-09-02T08:04:02.874488526-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = func(*args, **kwargs)
2025-09-02T08:04:02.874491666-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874494866-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874498246-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874502297-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874505317-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
2025-09-02T08:04:02.874508737-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = self.model_runner.execute_model(scheduler_output,
2025-09-02T08:04:02.874513247-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874516777-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874520167-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874524887-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874528187-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1733, in execute_model
2025-09-02T08:04:02.874531777-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     valid_sampled_token_ids = sampled_token_ids.tolist()
2025-09-02T08:04:02.874551317-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874554777-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] RuntimeError: CUDA error: an illegal memory access was encountered
2025-09-02T08:04:02.874558327-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-09-02T08:04:02.874561537-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-09-02T08:04:02.874564897-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-09-02T08:04:02.874568208-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.874572888-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Traceback (most recent call last):
2025-09-02T08:04:02.874576678-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 591, in worker_busy_loop
2025-09-02T08:04:02.874579908-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = func(*args, **kwargs)
2025-09-02T08:04:02.874588708-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874591738-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874636499-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874660099-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874664079-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
2025-09-02T08:04:02.874668569-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = self.model_runner.execute_model(scheduler_output,
2025-09-02T08:04:02.874671299-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874673619-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874676089-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874678519-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874681379-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1733, in execute_model
2025-09-02T08:04:02.874683709-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     valid_sampled_token_ids = sampled_token_ids.tolist()
2025-09-02T08:04:02.874687479-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874689709-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] RuntimeError: CUDA error: an illegal memory access was encountered
2025-09-02T08:04:02.874708630-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-09-02T08:04:02.874714670-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-09-02T08:04:02.874717670-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-09-02T08:04:02.874719940-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.874722800-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.874725130-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] WorkerProc hit an exception.
2025-09-02T08:04:02.874727330-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Traceback (most recent call last):
2025-09-02T08:04:02.874729610-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 591, in worker_busy_loop
2025-09-02T08:04:02.874731830-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = func(*args, **kwargs)
2025-09-02T08:04:02.874734060-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874739820-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874743060-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874745680-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874748450-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
2025-09-02T08:04:02.874751060-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = self.model_runner.execute_model(scheduler_output,
2025-09-02T08:04:02.874753520-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874755830-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874758040-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874760260-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874762470-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1733, in execute_model
2025-09-02T08:04:02.874764721-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     valid_sampled_token_ids = sampled_token_ids.tolist()
2025-09-02T08:04:02.874767181-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874769631-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] RuntimeError: CUDA error: an illegal memory access was encountered
2025-09-02T08:04:02.874775741-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-09-02T08:04:02.874778131-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-09-02T08:04:02.874780371-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-09-02T08:04:02.874782601-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.874784851-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Traceback (most recent call last):
2025-09-02T08:04:02.874787061-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 591, in worker_busy_loop
2025-09-02T08:04:02.874789471-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = func(*args, **kwargs)
2025-09-02T08:04:02.874792461-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874795311-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874797531-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874799741-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874801941-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
2025-09-02T08:04:02.874804201-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     output = self.model_runner.execute_model(scheduler_output,
2025-09-02T08:04:02.874806421-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874808711-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-09-02T08:04:02.874811031-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     return func(*args, **kwargs)
2025-09-02T08:04:02.874813331-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]            ^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874815641-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1733, in execute_model
2025-09-02T08:04:02.874817901-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]     valid_sampled_token_ids = sampled_token_ids.tolist()
2025-09-02T08:04:02.874820431-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.874822781-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] RuntimeError: CUDA error: an illegal memory access was encountered
2025-09-02T08:04:02.874825011-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-09-02T08:04:02.874827211-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-09-02T08:04:02.874838782-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-09-02T08:04:02.874841182-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.874843462-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m ERROR 09-02 08:04:02 [multiproc_executor.py:596]
2025-09-02T08:04:02.875067135-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.1.1) with config: model='Kwaipilot/KAT-V1-40B', speculative_config=None, tokenizer='Kwaipilot/KAT-V1-40B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=10000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=fmax, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null},
2025-09-02T08:04:02.876338404-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-6ac3a9cd9ebc4ce9b7e07eaff948f337,prompt_token_ids_len=969,mm_kwargs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=9031, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json={'properties': {'valuable': {'title': 'Valuable', 'type': 'boolean'}}, 'required': ['valuable'], 'title': 'IsFinanceRelatedNewsResponse', 'type': 'object', 'additionalProperties': False}, regex=None, choice=None, grammar=None, json_object=None, backend='xgrammar', backend_was_auto=True, disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, whitespace_pattern=None, structural_tag=None), extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 222, 223, 224, 225],),num_computed_tokens=912,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-d73229aee1504dcf88050970eff81b64'], resumed_from_preemption=[false], new_token_ids=[], new_block_ids=[[[]]], num_computed_tokens=[883]), num_scheduled_tokens={chatcmpl-d73229aee1504dcf88050970eff81b64: 1, chatcmpl-6ac3a9cd9ebc4ce9b7e07eaff948f337: 57}, total_num_scheduled_tokens=58, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={chatcmpl-6ac3a9cd9ebc4ce9b7e07eaff948f337: 1, chatcmpl-d73229aee1504dcf88050970eff81b64: 0}, grammar_bitmask=array([[        -1,         -1, 1073741823, ...,          0,          0,
2025-09-02T08:04:02.876390595-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:76]                  0],
2025-09-02T08:04:02.876399255-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:76]        [         0,          0,   67108864, ...,          0,          0,
2025-09-02T08:04:02.876402775-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:76]                  0]], shape=(2, 4752), dtype=int32), kv_connector_metadata=null)
2025-09-02T08:04:02.876715950-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.04584304584304588, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=969, hits=912), spec_decoding_stats=None, num_corrupted_reqs=0)
2025-09-02T08:04:02.878071901-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] EngineCore encountered a fatal error.
2025-09-02T08:04:02.878077631-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] Traceback (most recent call last):
2025-09-02T08:04:02.878080141-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 693, in run_engine_core
2025-09-02T08:04:02.878083591-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     engine_core.run_busy_loop()
2025-09-02T08:04:02.878085941-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 720, in run_busy_loop
2025-09-02T08:04:02.878088441-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     self._process_engine_step()
2025-09-02T08:04:02.878094261-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 745, in _process_engine_step
2025-09-02T08:04:02.878097631-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     outputs, model_executed = self.step_fn()
2025-09-02T08:04:02.878100741-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]                               ^^^^^^^^^^^^^^
2025-09-02T08:04:02.878103051-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 288, in step
2025-09-02T08:04:02.878105581-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     model_output = self.execute_model_with_error_logging(
2025-09-02T08:04:02.878108771-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.878111381-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 274, in execute_model_with_error_logging
2025-09-02T08:04:02.878113631-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     raise err
2025-09-02T08:04:02.878117162-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 265, in execute_model_with_error_logging
2025-09-02T08:04:02.878119442-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     return model_fn(scheduler_output)
2025-09-02T08:04:02.878121852-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.878129122-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 173, in execute_model
2025-09-02T08:04:02.878131512-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     (output, ) = self.collective_rpc(
2025-09-02T08:04:02.878133992-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]                  ^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.878136432-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 243, in collective_rpc
2025-09-02T08:04:02.878138622-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     result = get_response(w, dequeue_timeout)
2025-09-02T08:04:02.878140792-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.878143252-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 230, in get_response
2025-09-02T08:04:02.878146582-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702]     raise RuntimeError(
2025-09-02T08:04:02.878149902-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
2025-09-02T08:04:02.878153622-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2025-09-02T08:04:02.878157282-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
2025-09-02T08:04:02.878160872-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-09-02T08:04:02.878165372-07:00 �[1;36m(EngineCore_0 pid=309)�[0;0m ERROR 09-02 08:04:02 [core.py:702] ', please check the stack trace above for the root cause
2025-09-02T08:04:02.879742086-07:00 �[1;36m(VllmWorker TP0 pid=442)�[0;0m INFO 09-02 08:04:02 [multiproc_executor.py:520] Parent process exited, terminating worker
2025-09-02T08:04:02.880347276-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430] AsyncLLM output_handler failed.
2025-09-02T08:04:02.880362246-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430] Traceback (most recent call last):
2025-09-02T08:04:02.880368666-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 389, in output_handler
2025-09-02T08:04:02.880377956-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430]     outputs = await engine_core.get_output_async()
2025-09-02T08:04:02.880382636-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-02T08:04:02.880386946-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 843, in get_output_async
2025-09-02T08:04:02.880391306-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430]     raise self._format_exception(outputs) from None
2025-09-02T08:04:02.880395577-07:00 �[1;36m(APIServer pid=19)�[0;0m ERROR 09-02 08:04:02 [async_llm.py:430] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
2025-09-02T08:04:02.880593790-07:00 �[1;36m(VllmWorker TP1 pid=443)�[0;0m INFO 09-02 08:04:02 [multiproc_executor.py:520] Parent process exited, terminating worker
2025-09-02T08:04:02.880865794-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     100.64.0.27:60036 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2025-09-02T08:04:02.881339091-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     100.64.0.24:46412 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2025-09-02T08:04:02.916934453-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     Shutting down
2025-09-02T08:04:03.017513315-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     Waiting for application shutdown.
2025-09-02T08:04:03.017734188-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     Application shutdown complete.
2025-09-02T08:04:03.017802269-07:00 �[1;36m(APIServer pid=19)�[0;0m INFO:     Finished server process [19]
2025-09-02T08:04:03.152319187-07:00 Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
2025-09-02T08:04:03.157510416-07:00 Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:594 'an illegal memory access was encountered'
2025-09-02T08:04:04.870683630-07:00 nanobind: leaked 4 instances!
2025-09-02T08:04:04.870709711-07:00  - leaked instance 0x7fab59ea43d8 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
2025-09-02T08:04:04.870717661-07:00  - leaked instance 0x7fab59e8c828 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
2025-09-02T08:04:04.870720771-07:00  - leaked instance 0x7fab59e8f2e8 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
2025-09-02T08:04:04.870723521-07:00  - leaked instance 0x7fab59e8c8b8 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
2025-09-02T08:04:04.870726121-07:00 nanobind: leaked 2 types!
2025-09-02T08:04:04.870728811-07:00  - leaked type "xgrammar.xgrammar_bindings.GrammarMatcher"
2025-09-02T08:04:04.870733301-07:00  - leaked type "xgrammar.xgrammar_bindings.CompiledGrammar"
2025-09-02T08:04:04.870735901-07:00 nanobind: leaked 16 functions!
2025-09-02T08:04:04.870738541-07:00  - leaked function "reset"
2025-09-02T08:04:04.870741171-07:00  - leaked function "rollback"
2025-09-02T08:04:04.870744081-07:00  - leaked function "find_jump_forward_string"
2025-09-02T08:04:04.870748321-07:00  - leaked function ""
2025-09-02T08:04:04.870751781-07:00  - leaked function ""
2025-09-02T08:04:04.870754361-07:00  - leaked function "fill_next_token_bitmask"
2025-09-02T08:04:04.870757071-07:00  - leaked function "deserialize_json"
2025-09-02T08:04:04.870759931-07:00  - leaked function "serialize_json"
2025-09-02T08:04:04.870762541-07:00  - leaked function "_debug_print_internal_state"
2025-09-02T08:04:04.870765162-07:00  - leaked function "accept_token"
2025-09-02T08:04:04.870767862-07:00  - leaked function "is_terminated"
2025-09-02T08:04:04.870770462-07:00  - leaked function ""
2025-09-02T08:04:04.870773092-07:00  - leaked function ""
2025-09-02T08:04:04.870775642-07:00  - leaked function "accept_string"
2025-09-02T08:04:04.870778302-07:00  - leaked function ""
2025-09-02T08:04:04.870781062-07:00  - leaked function "__init__"
2025-09-02T08:04:04.870783712-07:00 nanobind: this is likely caused by a reference counting issue in the binding code.
2025-09-02T08:04:05.239477366-07:00 /usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown
2025-09-02T08:04:05.239533017-07:00   warnings.warn('resource_tracker: There appear to be %d '
2025-09-02T08:04:05.239539897-07:00 /usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
2025-09-02T08:04:05.239544697-07:00   warnings.warn('resource_tracker: There appear to be %d '

[njhill]

Yeah I'm guessing this has something to do with guided decoding / xgrammar

[Josephasafg]

@njhill Im facing the same issue and im not using guided decoding / xgrammar. Simply high concurrent load

[ggamsso]

I solved it by using guidance as the backend instead of xgrammer.

--guided-decoding-backend guidance --guided-decoding-disable-fallback

[ggamsso]

I solved it by using guidance as the backend instead of xgrammer.

--guided-decoding-backend guidance --guided-decoding-disable-fallback

Sorry, the method I suggested didn’t solve the issue.
It seems to occur when requests that use guided encoding and those that don’t are mixed together.

[hustxiayang]

@njhill It should be same as #19483?

[subnet-dev]

I'm encountering the same issue with vLLM 0.10.2 and the model Qwen/Qwen3-30B-A3B-Thinking-2507. I'm happy to run any additional tests or provide further details.

--model=Qwen/Qwen3-30B-A3B-Thinking-2507
--served-model-name=Qwen/Qwen3-30B-A3B-Thinking-2507
--max-model-len=131072
--reasoning-parser=deepseek_r1
--enable-auto-tool-choice
--tool-call-parser=hermes

Crash logs:

(APIServer pid=1) INFO 09-18 05:54:39 [loggers.py:123] Engine 000: Avg prompt throughput: 342.2 tokens/s, Avg generation throughput: 87.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 19.8%
(APIServer pid=1) INFO:     172.16.113.135:35006 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 05:54:49 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 104.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 19.8%
(APIServer pid=1) INFO 09-18 05:54:59 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 103.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 19.8%
(APIServer pid=1) INFO:     172.16.113.135:35700 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 05:55:09 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 101.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 19.8%
(APIServer pid=1) INFO:     172.16.113.135:39180 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 05:55:19 [loggers.py:123] Engine 000: Avg prompt throughput: 251.5 tokens/s, Avg generation throughput: 111.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 25.3%
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 09-18 05:55:29 [loggers.py:123] Engine 000: Avg prompt throughput: 342.6 tokens/s, Avg generation throughput: 158.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 23.2%
(APIServer pid=1) INFO:     172.16.113.135:59524 - "GET /metrics HTTP/1.1" 200 OK
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.2) with config: model='/mnt/models/hf/models--Qwen--Qwen3-30B-A3B-Thinking-2507/snapshots/144afc2f379b542fdd4e85a1fcd5e1f79112d95d', speculative_config=None, tokenizer='/mnt/models/hf/models--Qwen--Qwen3-30B-A3B-Thinking-2507/snapshots/144afc2f379b542fdd4e85a1fcd5e1f79112d95d', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-30B-A3B-Thinking-2507, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/mnt/models/cache/vllm/torch_compile_cache/ae9e2b04f2","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/mnt/models/cache/vllm/torch_compile_cache/ae9e2b04f2/rank_0_0/backbone"}, 
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0b890ef0-7d7c-43c6-b964-f6f23a18e034', 'chatcmpl-b0a9c0e7-0824-4ad3-aa07-1c22d45d8941'], resumed_from_preemption=[false, false], new_token_ids=[], new_block_ids=[null, null], num_computed_tokens=[3737, 4263]), num_scheduled_tokens={chatcmpl-b0a9c0e7-0824-4ad3-aa07-1c22d45d8941: 1, chatcmpl-0b890ef0-7d7c-43c6-b964-f6f23a18e034: 1}, total_num_scheduled_tokens=2, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], structured_output_request_ids={chatcmpl-b0a9c0e7-0824-4ad3-aa07-1c22d45d8941: 1, chatcmpl-0b890ef0-7d7c-43c6-b964-f6f23a18e034: 0}, grammar_bitmask=array([[       0,        0, 67108864, ...,        0,        0,        0],
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [dump_input.py:76]        [      -1,       -1,       -1, ...,       -1,       -1,       -1]],
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [dump_input.py:76]       shape=(2, 4748), dtype=int32), kv_connector_metadata=null)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.0529878371232152, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] Traceback (most recent call last):
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 711, in run_engine_core
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     self._process_engine_step()
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in _process_engine_step
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 292, in step
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 278, in execute_model_with_error_logging
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     raise err
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 269, in execute_model_with_error_logging
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     return model_fn(scheduler_output)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 93, in execute_model
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     return func(*args, **kwargs)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     return func(*args, **kwargs)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 436, in execute_model
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     return func(*args, **kwargs)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2129, in execute_model
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     ) = self._bookkeeping_sync(scheduler_output, sampler_output,
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1929, in _bookkeeping_sync
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     valid_sampled_token_ids = self._to_list(sampled_token_ids)
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3742, in _to_list
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     self.transfer_event.synchronize()
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/streams.py", line 231, in synchronize
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720]     super().synchronize()
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] torch.AcceleratorError: CUDA error: an illegal memory access was encountered
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=44) ERROR 09-18 05:55:31 [core.py:720] 
(EngineCore_DP0 pid=44) Process EngineCore_DP0:
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485] AsyncLLM output_handler failed.
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485] Traceback (most recent call last):
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in output_handler
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485]     outputs = await engine_core.get_output_async()
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 845, in get_output_async
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485]     raise self._format_exception(outputs) from None
(APIServer pid=1) ERROR 09-18 05:55:31 [async_llm.py:485] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore_DP0 pid=44) Traceback (most recent call last):
(APIServer pid=1) INFO:     172.16.5.127:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(EngineCore_DP0 pid=44)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=44)     self.run()
(EngineCore_DP0 pid=44)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=44)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 722, in run_engine_core
(EngineCore_DP0 pid=44)     raise e
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 711, in run_engine_core
(EngineCore_DP0 pid=44)     engine_core.run_busy_loop()
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop
(EngineCore_DP0 pid=44)     self._process_engine_step()
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in _process_engine_step
(EngineCore_DP0 pid=44)     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=44)                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 292, in step
(EngineCore_DP0 pid=44)     model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=44)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 278, in execute_model_with_error_logging
(EngineCore_DP0 pid=44)     raise err
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 269, in execute_model_with_error_logging
(EngineCore_DP0 pid=44)     return model_fn(scheduler_output)
(EngineCore_DP0 pid=44)            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 93, in execute_model
(EngineCore_DP0 pid=44)     output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=44)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=44)     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=44)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3060, in run_method
(EngineCore_DP0 pid=44)     return func(*args, **kwargs)
(EngineCore_DP0 pid=44)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=44)     return func(*args, **kwargs)
(EngineCore_DP0 pid=44)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 436, in execute_model
(EngineCore_DP0 pid=44)     output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=44)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=44)     return func(*args, **kwargs)
(EngineCore_DP0 pid=44)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2129, in execute_model
(EngineCore_DP0 pid=44)     ) = self._bookkeeping_sync(scheduler_output, sampler_output,
(EngineCore_DP0 pid=44)         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1929, in _bookkeeping_sync
(EngineCore_DP0 pid=44)     valid_sampled_token_ids = self._to_list(sampled_token_ids)
(EngineCore_DP0 pid=44)                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3742, in _to_list
(EngineCore_DP0 pid=44)     self.transfer_event.synchronize()
(EngineCore_DP0 pid=44)   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/streams.py", line 231, in synchronize
(EngineCore_DP0 pid=44)     super().synchronize()
(EngineCore_DP0 pid=44) torch.AcceleratorError: CUDA error: an illegal memory access was encountered
(EngineCore_DP0 pid=44) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=44) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=44) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=44) 
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
[rank0]:[W918 05:55:31.216900585 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) INFO:     172.16.4.170:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
nanobind: leaked 4 instances!
 - leaked instance 0x74e4e2835cc8 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x74e4e2836028 of type "xgrammar.xgrammar_bindings.CompiledGrammar"
 - leaked instance 0x74e4e27ad398 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked instance 0x74e4e27ae898 of type "xgrammar.xgrammar_bindings.GrammarMatcher"
nanobind: leaked 2 types!
 - leaked type "xgrammar.xgrammar_bindings.GrammarMatcher"
 - leaked type "xgrammar.xgrammar_bindings.CompiledGrammar"
nanobind: leaked 16 functions!
 - leaked function "fill_next_token_bitmask"
 - leaked function ""
 - leaked function ""
 - leaked function "is_terminated"
 - leaked function "accept_string"
 - leaked function "rollback"
 - leaked function "find_jump_forward_string"
 - leaked function "serialize_json"
 - leaked function "accept_token"
 - leaked function ""
 - leaked function "_debug_print_internal_state"
 - leaked function "deserialize_json"
 - leaked function "__init__"
 - leaked function ""
 - leaked function "reset"
 - leaked function ""
nanobind: this is likely caused by a reference counting issue in the binding code.
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.5.127:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.4.170:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.4.170:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.5.127:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.5.128:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     172.16.4.170:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     Shutting down
(APIServer pid=1) INFO:     Waiting for application shutdown.
(APIServer pid=1) INFO:     Application shutdown complete.
(APIServer pid=1) INFO:     Finished server process [1]

[stefanobranco]

Same happening here with 10.2 and various llama deployments (both 3 and 4). We can reproduce it quite consistently, I think it may have to do with concurrent requests that are both guided and not guided? We can try to narrow it down more if that helps.