[Bug]: Encountered AssertionError and precision issues when enabling MTP in deepseek v3.1

[WangQiangItachi]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.1.0
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.4.119-19.0009.56-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version        : 570.158.01
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   52 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          384
On-line CPU(s) list:             0-383
Vendor ID:                       AuthenticAMD
BIOS Vendor ID:                  Advanced Micro Devices, Inc.
Model name:                      AMD EPYC 9K84 96-Core Processor
BIOS Model name:                 AMD EPYC 9K84 96-Core Processor                
CPU family:                      25
Model:                           17
Thread(s) per core:              2
Core(s) per socket:              96
Socket(s):                       2
Stepping:                        1
Frequency boost:                 enabled
CPU max MHz:                     2600.0000
CPU min MHz:                     1500.0000
BogoMIPS:                        5200.95
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                  AMD-V
L1d cache:                       6 MiB (192 instances)
L1i cache:                       6 MiB (192 instances)
L2 cache:                        192 MiB (192 instances)
L3 cache:                        768 MiB (24 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-95,192-287
NUMA node1 CPU(s):               96-191,288-383
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.4.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.14.1
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==12.0.0
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0+cu128
[pip3] torchaudio==2.8.0+cu128
[pip3] torchvision==0.23.0+cu128
[pip3] transformers==4.57.0
[pip3] triton==3.4.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0rc2.dev403+g5bc26c438 (git sha: 5bc26c438)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     PHB     NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    PHB     PIX     NODE    SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-95,192-287    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    96-191,288-383  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    96-191,288-383  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PHB     96-191,288-383  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    PHB     PIX     96-191,288-383  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    PIX     PHB     NODE    SYS     SYS     SYS     SYS     NODE     X      PHB     NODE    SYS     SYS     SYS     SYS
NIC2    NODE    PHB     PIX     NODE    SYS     SYS     SYS     SYS     NODE    PHB      X      NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PHB     SYS     SYS     SYS     SYS     NODE    NODE     X      PHB
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    PHB     PIX     SYS     SYS     SYS     SYS     NODE    NODE    PHB      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

vllm serve

python3 -m vllm.entrypoints.cli.main serve DeepSeek-V3.1 \
     --tensor-parallel-size 16 \
     --trust-remote-code \
     --port 6000 \
     --max-num-seqs 16 \
     --max-num-batched-tokens 49152 \
     --max_model_len 49152 \
     --gpu-memory-utilization 0.9 \
     --enable-expert-parallel \
     --block-size 64 \
     --enable-chunked-prefill \
     --speculative-config '{"num_speculative_tokens": 1, "method": "deepseek_mtp"}'

AssertionError

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:02<00:00,  3.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:02<00:00,  3.34it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (decode, FULL):   0%|          | 0/6 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   0%|          | 0/6 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m WARNING 10-10 20:29:00 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=16,N=2048,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[128,128].json']
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m INFO 10-10 20:21:29 [gpu_worker.py:298] Available KV cache memory: 26.98 GiB�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] EngineCore failed to start.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     engine_core = EngineCoreProc(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 498, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     super().__init__(vllm_config, executor_class, log_stats,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 92, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._initialize_kv_caches(vllm_config)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 207, in _initialize_kv_caches
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self.model_executor.initialize_from_config(kv_cache_configs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self.collective_rpc("compile_or_warm_up_model")
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/executor_base.py", line 312, in collective_rpc
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return self._run_workers(method, *args, **(kwargs or {}))
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/ray_distributed_executor.py", line 505, in _run_workers
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     ray_worker_outputs = ray.get(ray_worker_outputs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return fn(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2882, in get
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 968, in get_objects
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     raise value.as_instanceof_cause()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] ray.exceptions.RayTaskError(AssertionError): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45582, ip=172.16.0.48, actor_id=99c37bbc4af57d1e9c3c27ea01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fe35d6c86b0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] AssertionError

After removing assert m.max_query_len <= self.reorder_batch_threshold in vllm/v1/attention/backends/mla/common.py, no errors occurred, but precision issues appeared

curl http://0.0.0.0:6000/v1/chat/completions  \
    -H "Content-Type: application/json"   \
    -d '{
    "model": "DeepSeek-V3.1",
    "messages": [
    {
        "role": "user",
        "content": "Where is the capital of the United States?"
    }
    ],
    "stream":false,
    "max_tokens": 30
  }'

output

{"id":"chatcmpl-2c1facdd85d84f109cf8ba4341ec7807","object":"chat.completion","created":1760163652,"model":"DeepSeek-V3.1","choices":[{"index":0,"message":{"role":"assistant","content":"Of\n  \"J.3-1, 4. The\n 0, and the\n 10. 8 2. The","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":43,"completion_tokens":30,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

curl http://0.0.0.0:6000/v1/chat/completions  \
    -H "Content-Type: application/json"   \
    -d '{
    "model": "DeepSeek-V3.1",
    "messages": [
    {
        "role": "user",
        "content": "深圳有哪些好玩的地方"
    }
    ],
    "stream":false,
    "max_tokens": 300
  }'

output

{"id":"chatcmpl-409d28efbe8245feaba037e9860185de","object":"chat.completion","created":1760162920,"model":"DeepSeek-V3.1","choices":[{"index":0,"message":{"role":"assistant","content":"当然，1, 0. 1,  | 0.0. The\n    |\n| 2, M. S. 0.0, 0, I’d\n\t\t\" type of the opposite of the same as the basis of the\n 6.4    | 0.0.1- 4, the\n\n    | 0.0; 1, 0.0, 0.0.0\n    - 1 0. This is a wide\nfrom the\n    | 0.0\n    - - - 1984-07-19, and is a\n 6. The two\n    | 0.0x1, 10. The one of the\n 3.1.1-15  \"wof:name\": \"C. The\n 1.2-22. 1. \"g,0000. The first\n// The process of a sharp at the\n * @param { \"id\": \"F = \"description\": \"https://www.education of the\n        if (m. 1. The\n     * <summary>\n    |    |  (2002. 29. 2010. 0.0.0.0) {\n        < 1991. 2009-1\n    | 0, and the\n    | 0.0.0.1","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":9,"total_tokens":309,"completion_tokens":300,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[benchislett]

Could you include the full engine output log, so I can check which attention backend is being used? Also, what hardware are you running on?

It's possible that this issue was either recently fixed or recently reintroduced. I can look into it

[WangQiangItachi]

Could you include the full engine output log, so I can check which attention backend is being used? Also, what hardware are you running on?

It's possible that this issue was either recently fixed or recently reintroduced. I can look into it

Here is the full engine output log, the attention backend is FlashMLA, and the hardware is H20.

/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from .version import __version__, __version_tuple__  # isort:skip
INFO 10-10 20:20:04 [__init__.py:216] Automatically detected platform cuda.
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:08 [api_server.py:1839] vLLM API server version dev
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:09 [utils.py:233] non-default args: {'model_tag': '/data0/models/deepseek-ai/DeepSeek-V3.1', 'port': 6000, 'model': '/data0/models/deepseek-ai/DeepSeek-V3.1', 'trust_remote_code': True, 'max_model_len': 49152, 'tensor_parallel_size': 16, 'enable_expert_parallel': True, 'block_size': 64, 'max_num_batched_tokens': 49152, 'max_num_seqs': 16, 'enable_chunked_prefill': True, 'speculative_config': {'num_speculative_tokens': 1, 'method': 'deepseek_mtp'}}
�[1;36m(APIServer pid=69250)�[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:09 [config.py:388] Replacing legacy 'type' key with 'rope_type'
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:09 [model.py:547] Resolved architecture: DeepseekV3ForCausalLM
�[1;36m(APIServer pid=69250)�[0;0m `torch_dtype` is deprecated! Use `dtype` instead!
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:09 [model.py:1510] Using max model len 49152
�[1;36m(APIServer pid=69250)�[0;0m WARNING 10-10 20:20:10 [speculative.py:214] method `deepseek_mtp` is deprecated and replaced with mtp.
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:10 [config.py:388] Replacing legacy 'type' key with 'rope_type'
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:10 [model.py:547] Resolved architecture: DeepSeekMTPModel
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:10 [model.py:1510] Using max model len 163840
�[1;36m(APIServer pid=69250)�[0;0m INFO 10-10 20:20:10 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=49152.
/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from .version import __version__, __version_tuple__  # isort:skip
INFO 10-10 20:20:13 [__init__.py:216] Automatically detected platform cuda.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:16 [core.py:644] Waiting for init message from front-end.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:16 [core.py:77] Initializing a V1 LLM engine (vdev) with config: model='/data0/models/deepseek-ai/DeepSeek-V3.1', speculative_config=SpeculativeConfig(method='mtp', model='/data0/models/deepseek-ai/DeepSeek-V3.1', num_spec_tokens=1), tokenizer='/data0/models/deepseek-ai/DeepSeek-V3.1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=49152, download_dir=None, load_format=auto, tensor_parallel_size=16, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/data0/models/deepseek-ai/DeepSeek-V3.1, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":32,"local_cache_dir":null}
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:20:16,864	INFO worker.py:1771 -- Connecting to existing Ray cluster at address: 172.16.0.44:1112...
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:20:16,874	INFO worker.py:1951 -- Connected to Ray cluster.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:29 [ray_utils.py:345] No current placement group found. Creating a new placement group.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m WARNING 10-10 20:20:29 [ray_utils.py:206] tensor_parallel_size=16 is bigger than a reserved number of GPUs (8 GPUs) in a node f022fe88372ba7c48fc3fc37780f2cfa1b39f964804c063a041c60da. Tensor parallel workers can be spread out to 2+ nodes which can degrade the performance unless you have fast interconnect across nodes, like Infiniband. To resolve this issue, make sure you have more than 16 GPUs available at each node.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m WARNING 10-10 20:20:29 [ray_utils.py:206] tensor_parallel_size=16 is bigger than a reserved number of GPUs (8 GPUs) in a node 7b4d2ff5074b262e83ec742f3c24cd0d7e65afb2cfd4765a8428e1ca. Tensor parallel workers can be spread out to 2+ nodes which can degrade the performance unless you have fast interconnect across nodes, like Infiniband. To resolve this issue, make sure you have more than 16 GPUs available at each node.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:29 [ray_distributed_executor.py:171] use_ray_spmd_worker: True
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(pid=69924)�[0m INFO 10-10 20:20:32 [__init__.py:216] Automatically detected platform cuda.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:37 [ray_env.py:63] RAY_NON_CARRY_OVER_ENV_VARS from config: set()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:37 [ray_env.py:65] Copying the following environment variables to workers: ['CUDA_HOME', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_USE_RAY_SPMD_WORKER', 'VLLM_USE_V1', 'VLLM_USE_RAY_COMPILED_DAG', 'VLLM_USAGE_SOURCE', 'LD_LIBRARY_PATH']
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:20:37 [ray_env.py:68] If certain env vars should NOT be copied, add them to /root/.config/vllm/ray_non_carry_over_env_vars.json file
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45579, ip=172.16.0.48)�[0m W1010 20:20:41.201000 45579 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45579, ip=172.16.0.48)�[0m W1010 20:20:41.201000 45579 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m W1010 20:20:41.444000 70644 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. �[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m W1010 20:20:41.444000 70644 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   1% Completed | 1/163 [00:00<00:20,  7.98it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   2% Completed | 4/163 [00:00<00:09, 17.09it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   4% Completed | 6/163 [00:00<00:11, 14.02it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   5% Completed | 8/163 [00:00<00:11, 13.14it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   7% Completed | 11/163 [00:00<00:09, 16.81it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   9% Completed | 14/163 [00:00<00:07, 19.28it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  10% Completed | 17/163 [00:01<00:10, 14.22it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  13% Completed | 21/163 [00:01<00:08, 15.82it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  14% Completed | 23/163 [00:01<00:09, 14.72it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  16% Completed | 26/163 [00:01<00:07, 17.51it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  18% Completed | 29/163 [00:01<00:06, 19.33it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  20% Completed | 32/163 [00:01<00:06, 20.87it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  21% Completed | 35/163 [00:02<00:06, 19.06it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  23% Completed | 38/163 [00:02<00:08, 15.27it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  25% Completed | 40/163 [00:02<00:08, 14.10it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  26% Completed | 42/163 [00:02<00:08, 13.65it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  27% Completed | 44/163 [00:02<00:10, 11.46it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  29% Completed | 47/163 [00:03<00:08, 14.03it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  30% Completed | 49/163 [00:03<00:08, 13.53it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  31% Completed | 51/163 [00:03<00:08, 12.84it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  33% Completed | 53/163 [00:03<00:08, 12.50it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  34% Completed | 55/163 [00:03<00:10, 10.76it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  36% Completed | 58/163 [00:03<00:07, 13.50it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  37% Completed | 60/163 [00:04<00:09, 11.34it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  38% Completed | 62/163 [00:04<00:08, 11.42it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  39% Completed | 64/163 [00:04<00:08, 11.32it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  40% Completed | 66/163 [00:04<00:07, 12.15it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  42% Completed | 68/163 [00:04<00:07, 12.02it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  43% Completed | 70/163 [00:05<00:07, 11.78it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  44% Completed | 72/163 [00:05<00:07, 11.80it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  45% Completed | 74/163 [00:05<00:07, 12.58it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  47% Completed | 77/163 [00:05<00:06, 13.29it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  48% Completed | 79/163 [00:05<00:06, 12.77it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  50% Completed | 81/163 [00:05<00:06, 12.64it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  51% Completed | 83/163 [00:06<00:06, 12.06it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  53% Completed | 86/163 [00:06<00:05, 13.50it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  54% Completed | 88/163 [00:06<00:05, 13.09it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  56% Completed | 92/163 [00:06<00:04, 15.08it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  58% Completed | 95/163 [00:06<00:04, 17.00it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  60% Completed | 97/163 [00:06<00:04, 15.30it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  61% Completed | 100/163 [00:07<00:03, 17.54it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  63% Completed | 102/163 [00:07<00:04, 13.43it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  64% Completed | 105/163 [00:07<00:04, 13.00it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  66% Completed | 107/163 [00:07<00:04, 12.26it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  67% Completed | 110/163 [00:07<00:04, 13.10it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  69% Completed | 112/163 [00:08<00:04, 11.08it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  71% Completed | 115/163 [00:08<00:03, 12.10it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  72% Completed | 117/163 [00:08<00:03, 12.13it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  74% Completed | 120/163 [00:08<00:02, 14.55it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  75% Completed | 122/163 [00:08<00:02, 14.09it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  77% Completed | 125/163 [00:08<00:02, 16.32it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  79% Completed | 128/163 [00:09<00:02, 15.64it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  80% Completed | 130/163 [00:09<00:02, 16.37it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  81% Completed | 132/163 [00:09<00:02, 15.28it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  82% Completed | 134/163 [00:09<00:02, 14.25it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  84% Completed | 137/163 [00:09<00:01, 16.60it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  85% Completed | 139/163 [00:09<00:01, 15.10it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  87% Completed | 141/163 [00:10<00:01, 14.10it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  88% Completed | 143/163 [00:10<00:01, 11.80it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  89% Completed | 145/163 [00:10<00:01, 11.88it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  91% Completed | 148/163 [00:10<00:01, 14.62it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  92% Completed | 150/163 [00:10<00:00, 14.08it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  94% Completed | 153/163 [00:10<00:00, 15.11it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  95% Completed | 155/163 [00:11<00:00, 11.97it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  97% Completed | 158/163 [00:11<00:00, 14.54it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  99% Completed | 162/163 [00:11<00:00, 18.57it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:11<00:00, 14.15it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:   8% Completed | 13/163 [00:00<00:01, 129.32it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  17% Completed | 28/163 [00:00<00:00, 137.60it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  26% Completed | 42/163 [00:00<00:00, 137.05it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  34% Completed | 56/163 [00:00<00:00, 133.73it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  43% Completed | 70/163 [00:00<00:00, 134.22it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  52% Completed | 85/163 [00:00<00:00, 136.03it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  61% Completed | 99/163 [00:00<00:00, 111.05it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  69% Completed | 113/163 [00:00<00:00, 116.82it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  77% Completed | 126/163 [00:01<00:00, 117.27it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  85% Completed | 139/163 [00:01<00:00, 119.56it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards:  93% Completed | 152/163 [00:01<00:00, 98.11it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m  
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:01<00:00, 97.72it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:01<00:00, 114.37it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m [Gloo] Rank 3 is connected to 15 peer ranks. Expected number of connected peer ranks is : 15
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m [Gloo] Rank 3 is connected to 15 peer ranks. Expected number of connected peer ranks is : 15
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m INFO 10-10 20:20:43 [__init__.py:1384] Found nccl from library libnccl.so.2
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m INFO 10-10 20:20:43 [pynccl.py:103] vLLM is using nccl==2.27.3
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:20:33 [__init__.py:216] Automatically detected platform cuda.�[32m [repeated 15x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m WARNING 10-10 20:20:46 [symm_mem.py:66] SymmMemCommunicator: World size 16 not supported, communicator is not available.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m WARNING 10-10 20:20:46 [custom_all_reduce.py:86] Custom allreduce is disabled because this process group spans across nodes.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m INFO 10-10 20:20:46 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_03f84e23'), local_subscribe_addr='ipc:///tmp/1943c79f-7778-4f22-8138-fab0f93fc311', remote_subscribe_addr='tcp://172.16.0.44:35447', remote_addr_ipv6=False)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m INFO 10-10 20:20:47 [parallel_state.py:1208] rank 3 in world size 16 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m INFO 10-10 20:20:47 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m INFO 10-10 20:20:48 [gpu_model_runner.py:2602] Starting to load model /data0/models/deepseek-ai/DeepSeek-V3.1...
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m INFO 10-10 20:20:48 [gpu_model_runner.py:2634] Loading model from scratch...
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m INFO 10-10 20:20:48 [cuda.py:288] Using FlashMLA backend on V1 engine.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m INFO 10-10 20:20:48 [layer.py:1052] [EP Rank 13/16] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 16/256. Experts local to global index map: 0->208, 1->209, 2->210, 3->211, 4->212, 5->213, 6->214, 7->215, 8->216, 9->217, 10->218, 11->219, 12->220, 13->221, 14->222, 15->223.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m INFO 10-10 20:20:48 [fp8.py:462] Using DeepGemm kernels for Fp8MoEMethod.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m WARNING 10-10 20:20:48 [fp8.py:480] CutlassBlockScaledGroupedGemm not supported on the current platform.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m [Gloo] Rank 15 is connected to 15 peer ranks. Expected number of connected peer ranks is : 15�[32m [repeated 94x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:20:46 [__init__.py:1384] Found nccl from library libnccl.so.2�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:20:46 [pynccl.py:103] vLLM is using nccl==2.27.3�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:21:00 [default_loader.py:267] Loading weights took 11.02 seconds
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:21:00 [deep_gemm.py:59] DeepGEMM E8M0 disabled on current configuration.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m WARNING 10-10 20:20:46 [symm_mem.py:66] SymmMemCommunicator: World size 16 not supported, communicator is not available.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m WARNING 10-10 20:20:46 [custom_all_reduce.py:86] Custom allreduce is disabled because this process group spans across nodes.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:20:47 [parallel_state.py:1208] rank 15 in world size 16 is assigned as DP rank 0, PP rank 0, TP rank 15, EP rank 15�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:20:48 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m INFO 10-10 20:20:48 [gpu_model_runner.py:2602] Starting to load model /data0/models/deepseek-ai/DeepSeek-V3.1...�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m INFO 10-10 20:20:49 [gpu_model_runner.py:2634] Loading model from scratch...�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:20:49 [cuda.py:288] Using FlashMLA backend on V1 engine.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:20:49 [layer.py:1052] [EP Rank 1/16] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 16/256. Experts local to global index map: 0->16, 1->17, 2->18, 3->19, 4->20, 5->21, 6->22, 7->23, 8->24, 9->25, 10->26, 11->27, 12->28, 13->29, 14->30, 15->31.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:20:49 [fp8.py:462] Using DeepGemm kernels for Fp8MoEMethod.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m WARNING 10-10 20:20:49 [fp8.py:480] CutlassBlockScaledGroupedGemm not supported on the current platform.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m INFO 10-10 20:21:00 [gpu_model_runner.py:2641] Loading drafter model...
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m WARNING 10-10 20:21:00 [__init__.py:732] `torch.compile` is turned on, but the model /data0/models/deepseek-ai/DeepSeek-V3.1 does not support it. Please open an issue on GitHub if you want it to be supported.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m INFO 10-10 20:21:01 [eagle.py:869] Assuming the EAGLE head shares the same vocab embedding with the target model.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m INFO 10-10 20:21:01 [eagle.py:888] Loading EAGLE LM head weights from the target model.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:21:02 [gpu_model_runner.py:2653] Model loading took 41.3170 GiB and 12.983501 seconds
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m INFO 10-10 20:21:10 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/df07acdee4/rank_4_0/backbone for vLLM's torch.compile
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m INFO 10-10 20:21:10 [backends.py:559] Dynamo bytecode transform time: 6.80 s
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:03 [default_loader.py:267] Loading weights took 1.38 seconds�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:01 [deep_gemm.py:59] DeepGEMM E8M0 disabled on current configuration.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:01 [gpu_model_runner.py:2641] Loading drafter model...�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m WARNING 10-10 20:21:01 [__init__.py:732] `torch.compile` is turned on, but the model /data0/models/deepseek-ai/DeepSeek-V3.1 does not support it. Please open an issue on GitHub if you want it to be supported.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:03 [eagle.py:869] Assuming the EAGLE head shares the same vocab embedding with the target model.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:03 [eagle.py:888] Loading EAGLE LM head weights from the target model.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m INFO 10-10 20:21:03 [gpu_model_runner.py:2653] Model loading took 41.3170 GiB and 14.013194 seconds�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m INFO 10-10 20:21:14 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 2.788 s
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m INFO 10-10 20:21:23 [monitor.py:34] torch.compile takes 7.05 s in total
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:21:11 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/df07acdee4/rank_1_0/backbone for vLLM's torch.compile�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:21:11 [backends.py:559] Dynamo bytecode transform time: 7.13 s�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m INFO 10-10 20:21:14 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 2.985 s�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m INFO 10-10 20:21:28 [gpu_worker.py:298] Available KV cache memory: 26.98 GiB
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m INFO 10-10 20:21:23 [monitor.py:34] torch.compile takes 6.97 s in total�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1087] GPU KV cache size: 405,632 tokens
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:21:29 [kv_cache_utils.py:1091] Maximum concurrency for 49,152 tokens per request: 8.25x
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):   1%|          | 513/49152 [00:00<00:42, 1156.21it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):   9%|▉         | 4320/49152 [00:03<00:36, 1244.98it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):   9%|▉         | 4445/49152 [00:03<00:35, 1245.14it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  15%|█▍        | 7267/49152 [00:05<00:31, 1323.83it/s]�[32m [repeated 660x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  21%|██        | 10436/49152 [00:07<00:27, 1426.62it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  22%|██▏       | 10580/49152 [00:07<00:26, 1429.64it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  30%|██▉       | 14520/49152 [00:10<00:22, 1562.76it/s]�[32m [repeated 773x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45579, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  30%|██▉       | 14614/49152 [00:10<00:21, 1574.77it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  30%|███       | 14772/49152 [00:10<00:21, 1575.50it/s]�[32m [repeated 5x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  48%|████▊     | 23799/49152 [00:15<00:11, 2121.18it/s]�[32m [repeated 721x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  61%|██████    | 29883/49152 [00:18<00:07, 2654.98it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  61%|██████▏   | 30168/49152 [00:18<00:06, 2712.60it/s]�[32m [repeated 27x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  77%|███████▋  | 37646/49152 [00:20<00:03, 3366.42it/s]�[32m [repeated 704x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  93%|█████████▎| 45569/49152 [00:21<00:00, 7682.13it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  96%|█████████▌| 47244/49152 [00:21<00:00, 10093.37it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])): 100%|██████████| 49152/49152 [00:21<00:00, 2252.92it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   0%|          | 0/49152 [00:00<?, ?it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   2%|▏         | 739/49152 [00:00<00:06, 7381.42it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   8%|▊         | 3929/49152 [00:01<00:17, 2547.16it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   9%|▊         | 4186/49152 [00:01<00:17, 2546.26it/s]�[32m [repeated 20x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  20%|█▉        | 9712/49152 [00:03<00:14, 2799.85it/s]�[32m [repeated 598x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])): 100%|██████████| 49152/49152 [00:21<00:00, 2239.75it/s] �[32m [repeated 43x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 14x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45581, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])):  96%|█████████▌| 47233/49152 [00:21<00:00, 10014.09it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([1536, 1536])): 100%|██████████| 49152/49152 [00:21<00:00, 2241.71it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  38%|███▊      | 18689/49152 [00:06<00:08, 3515.54it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  39%|███▉      | 19059/49152 [00:06<00:08, 3568.41it/s]�[32m [repeated 29x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  60%|██████    | 29506/49152 [00:08<00:03, 5258.72it/s]�[32m [repeated 714x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  92%|█████████▏| 45030/49152 [00:10<00:00, 13316.24it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  96%|█████████▌| 47204/49152 [00:10<00:00, 15814.43it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])): 100%|██████████| 49152/49152 [00:10<00:00, 4556.86it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  82%|████████▏ | 40193/49152 [00:10<00:01, 7700.45it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  85%|████████▌ | 41885/49152 [00:10<00:00, 10286.91it/s]�[32m [repeated 27x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45581, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   3%|▎         | 1596/49152 [00:02<01:42, 464.76it/s]�[32m [repeated 355x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])): 100%|██████████| 49152/49152 [00:10<00:00, 4519.96it/s] �[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])):  96%|█████████▌| 47164/49152 [00:10<00:00, 15652.10it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2048, 512])): 100%|██████████| 49152/49152 [00:10<00:00, 4520.08it/s] �[32m [repeated 5x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   6%|▌         | 2841/49152 [00:06<02:00, 384.96it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   6%|▌         | 2880/49152 [00:06<02:00, 385.57it/s]�[32m [repeated 14x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):   7%|▋         | 3580/49152 [00:08<01:55, 395.78it/s]�[32m [repeated 695x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  10%|▉         | 4861/49152 [00:11<01:49, 404.58it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  10%|▉         | 4902/49152 [00:11<01:49, 405.55it/s]�[32m [repeated 22x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  11%|█▏        | 5629/49152 [00:13<01:45, 412.29it/s]�[32m [repeated 753x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  14%|█▍        | 7011/49152 [00:16<01:38, 426.27it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  14%|█▍        | 7054/49152 [00:16<01:38, 426.79it/s]�[32m [repeated 29x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  16%|█▌        | 7663/49152 [00:17<01:35, 432.94it/s]�[32m [repeated 733x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  18%|█▊        | 9083/49152 [00:21<01:29, 446.75it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  19%|█▊        | 9128/49152 [00:21<01:29, 446.99it/s]�[32m [repeated 30x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  20%|██        | 9873/49152 [00:22<01:25, 458.83it/s]�[32m [repeated 751x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  23%|██▎       | 11315/49152 [00:26<01:20, 469.52it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  23%|██▎       | 11363/49152 [00:26<01:20, 470.47it/s]�[32m [repeated 27x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  25%|██▍       | 12264/49152 [00:27<01:15, 485.48it/s]�[32m [repeated 726x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  29%|██▊       | 14036/49152 [00:31<01:08, 509.06it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  29%|██▊       | 14088/49152 [00:31<01:08, 509.67it/s]�[32m [repeated 33x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  30%|███       | 14820/49152 [00:32<01:06, 519.48it/s]�[32m [repeated 739x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  34%|███▍      | 16783/49152 [00:36<00:58, 550.40it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  34%|███▍      | 16839/49152 [00:36<00:58, 551.21it/s]�[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  36%|███▌      | 17670/49152 [00:38<00:55, 563.79it/s]�[32m [repeated 726x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  40%|███▉      | 19601/49152 [00:41<00:49, 600.84it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  40%|████      | 19662/49152 [00:41<00:48, 601.93it/s]�[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  42%|████▏     | 20678/49152 [00:43<00:45, 623.54it/s]�[32m [repeated 730x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  46%|████▋     | 22811/49152 [00:46<00:39, 668.01it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  47%|████▋     | 22879/49152 [00:46<00:39, 670.28it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  49%|████▊     | 23890/49152 [00:48<00:36, 697.21it/s]�[32m [repeated 733x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  53%|█████▎    | 26114/49152 [00:51<00:30, 754.64it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  53%|█████▎    | 26191/49152 [00:51<00:30, 757.07it/s]�[32m [repeated 25x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  56%|█████▋    | 27704/49152 [00:53<00:26, 813.44it/s]�[32m [repeated 745x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45581, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  63%|██████▎   | 31072/49152 [00:56<00:18, 952.48it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  63%|██████▎   | 31169/49152 [00:56<00:18, 956.78it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  66%|██████▌   | 32498/49152 [00:58<00:16, 1029.50it/s]�[32m [repeated 733x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  74%|███████▍  | 36382/49152 [01:01<00:09, 1290.51it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  74%|███████▍  | 36516/49152 [01:02<00:09, 1302.99it/s]�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  78%|███████▊  | 38544/49152 [01:03<00:06, 1631.05it/s]�[32m [repeated 682x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  91%|█████████▏| 44966/49152 [01:05<00:01, 3215.83it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  92%|█████████▏| 45344/49152 [01:05<00:01, 3369.84it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  93%|█████████▎| 45699/49152 [01:05<00:01, 2795.85it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  93%|█████████▎| 45690/49152 [01:06<00:01, 2752.03it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  95%|█████████▍| 46561/49152 [01:06<00:00, 4229.99it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  89%|████████▉ | 43777/49152 [01:05<00:03, 1730.50it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  91%|█████████ | 44603/49152 [01:06<00:01, 3178.65it/s]�[32m [repeated 26x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   2%|▏         | 1038/49152 [00:03<02:44, 292.32it/s]�[32m [repeated 401x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  97%|█████████▋| 47503/49152 [01:07<00:00, 4291.51it/s]�[32m [repeated 97x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])):  99%|█████████▉| 48642/49152 [01:07<00:00, 6184.95it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1024])): 100%|██████████| 49152/49152 [01:07<00:00, 730.17it/s] �[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   4%|▍         | 1866/49152 [00:07<04:26, 177.51it/s]�[32m [repeated 257x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   4%|▍         | 1884/49152 [00:07<04:26, 177.14it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   4%|▍         | 1902/49152 [00:07<04:27, 176.83it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   5%|▌         | 2693/49152 [00:12<04:18, 179.89it/s]�[32m [repeated 726x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   6%|▌         | 2705/49152 [00:12<04:19, 179.04it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   6%|▌         | 2723/49152 [00:12<04:19, 179.04it/s]�[32m [repeated 26x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   7%|▋         | 3679/49152 [00:17<04:06, 184.35it/s]�[32m [repeated 739x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   7%|▋         | 3681/49152 [00:17<04:06, 184.59it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   8%|▊         | 3700/49152 [00:17<04:06, 184.60it/s]�[32m [repeated 20x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   9%|▉         | 4610/49152 [00:22<03:55, 188.81it/s]�[32m [repeated 757x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   9%|▉         | 4610/49152 [00:22<03:55, 188.97it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):   9%|▉         | 4629/49152 [00:22<03:55, 188.98it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  11%|█▏        | 5532/49152 [00:27<03:45, 193.52it/s]�[32m [repeated 720x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  11%|█         | 5510/49152 [00:27<03:45, 193.32it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  11%|█▏        | 5530/49152 [00:27<03:45, 193.45it/s]�[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  13%|█▎        | 6550/49152 [00:32<03:35, 198.14it/s]�[32m [repeated 751x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  13%|█▎        | 6570/49152 [00:32<03:34, 198.45it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  13%|█▎        | 6590/49152 [00:33<03:34, 198.52it/s]�[32m [repeated 19x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  15%|█▌        | 7512/49152 [00:37<03:29, 198.81it/s]�[32m [repeated 730x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  15%|█▌        | 7570/49152 [00:38<03:29, 198.61it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  15%|█▌        | 7590/49152 [00:38<03:29, 198.62it/s]�[32m [repeated 32x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  17%|█▋        | 8533/49152 [00:42<03:19, 203.99it/s]�[32m [repeated 737x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  17%|█▋        | 8552/49152 [00:42<03:19, 203.93it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  17%|█▋        | 8573/49152 [00:42<03:18, 203.93it/s]�[32m [repeated 21x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  20%|█▉        | 9602/49152 [00:47<03:09, 209.21it/s]�[32m [repeated 729x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  20%|█▉        | 9646/49152 [00:48<03:08, 209.49it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  20%|█▉        | 9667/49152 [00:48<03:08, 209.48it/s]�[32m [repeated 33x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  21%|██        | 10256/49152 [00:53<23:21, 27.75it/s] �[32m [repeated 486x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  20%|██        | 10003/49152 [00:50<03:03, 213.11it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  20%|██        | 10025/49152 [00:50<03:03, 213.13it/s]�[32m [repeated 9x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  24%|██▍       | 11832/49152 [00:58<02:48, 221.19it/s]�[32m [repeated 383x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  24%|██▍       | 11741/49152 [00:57<02:49, 221.32it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  24%|██▍       | 11764/49152 [00:57<02:48, 221.28it/s]�[32m [repeated 4x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  26%|██▋       | 12935/49152 [01:02<02:38, 227.91it/s]�[32m [repeated 749x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  26%|██▋       | 12983/49152 [01:03<02:38, 227.82it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  26%|██▋       | 13006/49152 [01:03<02:38, 227.82it/s]�[32m [repeated 22x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  29%|██▊       | 14106/49152 [01:07<02:29, 234.79it/s]�[32m [repeated 734x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  29%|██▊       | 14059/49152 [01:07<02:29, 234.44it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  29%|██▊       | 14083/49152 [01:07<02:29, 234.43it/s]�[32m [repeated 27x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  31%|███       | 15347/49152 [01:13<02:19, 241.86it/s]�[32m [repeated 743x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  31%|███▏      | 15373/49152 [01:13<02:18, 243.17it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  31%|███▏      | 15398/49152 [01:13<02:17, 245.08it/s]�[32m [repeated 21x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  34%|███▎      | 16579/49152 [01:18<02:06, 257.63it/s]�[32m [repeated 725x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  34%|███▎      | 16497/49152 [01:18<02:08, 254.67it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  34%|███▎      | 16523/49152 [01:18<02:07, 255.19it/s]�[32m [repeated 38x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  36%|███▋      | 17888/49152 [01:23<01:57, 266.62it/s]�[32m [repeated 712x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  37%|███▋      | 17971/49152 [01:23<01:56, 266.82it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  37%|███▋      | 17998/49152 [01:23<01:56, 266.86it/s]�[32m [repeated 37x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  39%|███▉      | 19231/49152 [01:28<01:48, 276.57it/s]�[32m [repeated 732x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  39%|███▉      | 19317/49152 [01:28<01:47, 276.28it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  39%|███▉      | 19345/49152 [01:28<01:47, 276.28it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  42%|████▏     | 20693/49152 [01:33<01:39, 286.88it/s]�[32m [repeated 720x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  42%|████▏     | 20811/49152 [01:33<01:38, 286.78it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  42%|████▏     | 20840/49152 [01:33<01:38, 286.78it/s]�[32m [repeated 41x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  45%|████▌     | 22189/49152 [01:38<01:27, 308.09it/s]�[32m [repeated 746x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45580, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  45%|████▌     | 22162/49152 [01:38<01:27, 307.49it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  45%|████▌     | 22193/49152 [01:38<01:27, 308.15it/s]�[32m [repeated 24x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  48%|████▊     | 23784/49152 [01:43<01:18, 322.52it/s]�[32m [repeated 748x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  48%|████▊     | 23514/49152 [01:43<01:20, 320.25it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  48%|████▊     | 23547/49152 [01:43<01:19, 320.27it/s]�[32m [repeated 26x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  52%|█████▏    | 25512/49152 [01:48<01:07, 348.44it/s]�[32m [repeated 750x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  52%|█████▏    | 25477/49152 [01:48<01:08, 347.48it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  52%|█████▏    | 25513/49152 [01:48<01:07, 348.68it/s]�[32m [repeated 25x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  55%|█████▌    | 27153/49152 [01:53<00:59, 367.56it/s]�[32m [repeated 735x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  55%|█████▌    | 27264/49152 [01:53<00:59, 367.92it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  56%|█████▌    | 27301/49152 [01:53<00:59, 367.93it/s]�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  60%|█████▉    | 29304/49152 [01:58<00:48, 406.83it/s]�[32m [repeated 745x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  60%|█████▉    | 29301/49152 [01:58<00:48, 406.97it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  60%|█████▉    | 29342/49152 [01:58<00:48, 407.01it/s]�[32m [repeated 27x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  64%|██████▎   | 31282/49152 [02:03<00:37, 479.43it/s]�[32m [repeated 472x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  61%|██████    | 29787/49152 [02:00<00:46, 414.92it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  61%|██████    | 29830/49152 [02:01<00:46, 418.05it/s]�[32m [repeated 16x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  69%|██████▉   | 33796/49152 [02:08<00:29, 514.83it/s]�[32m [repeated 694x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  68%|██████▊   | 33409/49152 [02:08<00:30, 509.00it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  68%|██████▊   | 33461/49152 [02:08<00:30, 509.68it/s]�[32m [repeated 24x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  75%|███████▍  | 36664/49152 [02:13<00:19, 633.05it/s]�[32m [repeated 715x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  75%|███████▍  | 36799/49152 [02:13<00:19, 638.53it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  75%|███████▌  | 36864/49152 [02:13<00:19, 639.79it/s]�[32m [repeated 33x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45580, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  82%|████████▏ | 40112/49152 [02:18<00:10, 841.26it/s]�[32m [repeated 553x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  81%|████████  | 39847/49152 [02:17<00:11, 812.27it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  81%|████████  | 39933/49152 [02:18<00:11, 825.05it/s]�[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  92%|█████████▏| 45234/49152 [02:22<00:02, 1747.03it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  93%|█████████▎| 45528/49152 [02:22<00:02, 1418.74it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  93%|█████████▎| 45762/49152 [02:22<00:02, 1483.49it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  94%|█████████▍| 46306/49152 [02:22<00:01, 2165.69it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  95%|█████████▍| 46620/49152 [02:22<00:01, 1677.99it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  96%|█████████▌| 46977/49152 [02:23<00:01, 1928.69it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  97%|█████████▋| 47489/49152 [02:23<00:00, 2365.33it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])): 100%|█████████▉| 48932/49152 [02:23<00:00, 4318.68it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])): 100%|██████████| 49152/49152 [02:23<00:00, 342.76it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  88%|████████▊ | 43192/49152 [02:22<00:05, 1152.68it/s]�[32m [repeated 457x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  88%|████████▊ | 43327/49152 [02:22<00:04, 1188.78it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  88%|████████▊ | 43454/49152 [02:22<00:04, 1211.79it/s]�[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  98%|█████████▊| 48001/49152 [02:25<00:00, 2880.46it/s]�[32m [repeated 121x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])):  99%|█████████▉| 48897/49152 [02:25<00:00, 4204.96it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([2304, 7168])): 100%|██████████| 49152/49152 [02:25<00:00, 337.47it/s] �[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   4%|▍         | 2010/49152 [00:04<02:12, 356.55it/s]�[32m [repeated 197x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   4%|▍         | 2117/49152 [00:04<02:14, 350.87it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   4%|▍         | 2153/49152 [00:04<02:14, 350.07it/s]�[32m [repeated 3x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   8%|▊         | 3805/49152 [00:09<02:06, 358.64it/s]�[32m [repeated 723x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   6%|▋         | 3159/49152 [00:07<02:10, 351.59it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):   7%|▋         | 3195/49152 [00:07<02:10, 351.96it/s]�[32m [repeated 23x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  12%|█▏        | 5669/49152 [00:14<01:56, 373.82it/s]�[32m [repeated 729x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  11%|█▏        | 5549/49152 [00:14<01:57, 372.45it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  11%|█▏        | 5587/49152 [00:14<01:56, 372.47it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  15%|█▌        | 7575/49152 [00:19<01:46, 391.34it/s]�[32m [repeated 728x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  15%|█▌        | 7571/49152 [00:19<01:46, 390.80it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  15%|█▌        | 7611/49152 [00:19<01:46, 391.06it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  20%|█▉        | 9596/49152 [00:24<01:36, 410.93it/s]�[32m [repeated 730x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  20%|█▉        | 9769/49152 [00:24<01:35, 413.37it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  20%|█▉        | 9811/49152 [00:25<01:34, 414.23it/s]�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  24%|██▍       | 11709/49152 [00:29<01:26, 434.94it/s]�[32m [repeated 728x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  24%|██▍       | 11965/49152 [00:30<01:25, 435.51it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  24%|██▍       | 12009/49152 [00:30<01:25, 435.55it/s]�[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  28%|██▊       | 13974/49152 [00:34<01:16, 460.21it/s]�[32m [repeated 733x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  29%|██▉       | 14243/49152 [00:35<01:15, 464.31it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  29%|██▉       | 14290/49152 [00:35<01:15, 464.78it/s]�[32m [repeated 34x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  33%|███▎      | 16397/49152 [00:39<01:06, 491.06it/s]�[32m [repeated 734x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45581, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  34%|███▍      | 16602/49152 [00:40<01:05, 493.29it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  34%|███▍      | 16652/49152 [00:40<01:05, 493.99it/s]�[32m [repeated 33x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  39%|███▊      | 18993/49152 [00:44<00:56, 534.30it/s]�[32m [repeated 715x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  37%|███▋      | 17980/49152 [00:43<01:00, 512.05it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  37%|███▋      | 18032/49152 [00:43<01:00, 512.47it/s]�[32m [repeated 42x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  44%|████▍     | 21839/49152 [00:49<00:46, 586.91it/s]�[32m [repeated 730x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  45%|████▍     | 22024/49152 [00:50<00:46, 588.15it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  45%|████▍     | 22084/49152 [00:50<00:45, 590.42it/s]�[32m [repeated 35x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  51%|█████     | 24979/49152 [00:54<00:36, 657.72it/s]�[32m [repeated 742x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  48%|████▊     | 23673/49152 [00:53<00:41, 621.00it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  48%|████▊     | 23736/49152 [00:53<00:40, 621.38it/s]�[32m [repeated 26x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  58%|█████▊    | 28562/49152 [00:59<00:26, 764.53it/s]�[32m [repeated 729x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45579, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  59%|█████▊    | 28796/49152 [01:00<00:26, 774.29it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  59%|█████▊    | 28875/49152 [01:00<00:26, 777.41it/s]�[32m [repeated 36x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  67%|██████▋   | 32878/49152 [01:04<00:17, 948.34it/s]�[32m [repeated 745x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  63%|██████▎   | 30870/49152 [01:03<00:21, 846.35it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  63%|██████▎   | 30956/49152 [01:03<00:21, 847.67it/s]�[32m [repeated 31x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  79%|███████▉  | 38838/49152 [01:09<00:07, 1464.96it/s]�[32m [repeated 690x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  80%|████████  | 39505/49152 [01:10<00:06, 1506.90it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  81%|████████  | 39663/49152 [01:10<00:06, 1526.13it/s]�[32m [repeated 28x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  91%|█████████▏| 44923/49152 [01:12<00:01, 2947.67it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  92%|█████████▏| 45261/49152 [01:12<00:01, 3057.23it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  93%|█████████▎| 45596/49152 [01:12<00:01, 2369.47it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  95%|█████████▍| 46507/49152 [01:12<00:00, 3887.18it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  96%|█████████▌| 46976/49152 [01:13<00:00, 3852.71it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  96%|█████████▋| 47418/49152 [01:13<00:00, 3844.92it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  99%|█████████▊| 48476/49152 [01:13<00:00, 5553.38it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])): 100%|█████████▉| 49090/49152 [01:13<00:00, 4469.36it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])): 100%|██████████| 49152/49152 [01:13<00:00, 668.66it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):   2%|▏         | 1175/49152 [00:00<00:15, 3027.94it/s]�[32m [repeated 490x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  87%|████████▋ | 42842/49152 [01:12<00:03, 2090.90it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  88%|████████▊ | 43072/49152 [01:12<00:02, 2150.95it/s]�[32m [repeated 18x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  95%|█████████▍| 46504/49152 [01:14<00:00, 3816.94it/s]�[32m [repeated 101x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])): 100%|█████████▉| 48978/49152 [01:14<00:00, 4304.65it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])): 100%|██████████| 49152/49152 [01:14<00:00, 657.85it/s] �[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  20%|█▉        | 9595/49152 [00:05<00:24, 1608.55it/s]�[32m [repeated 370x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70306)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  20%|█▉        | 9789/49152 [00:06<00:24, 1611.18it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  20%|██        | 9978/49152 [00:06<00:23, 1669.93it/s]�[32m [repeated 10x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 1152])):  98%|█████████▊| 48390/49152 [01:14<00:00, 5278.16it/s]�[32m [repeated 8x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  39%|███▉      | 19141/49152 [00:11<00:14, 2133.89it/s]�[32m [repeated 639x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  41%|████      | 19935/49152 [00:11<00:13, 2110.75it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  41%|████      | 20186/49152 [00:11<00:13, 2204.38it/s]�[32m [repeated 32x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  65%|██████▌   | 32159/49152 [00:16<00:04, 3564.46it/s]�[32m [repeated 642x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  51%|█████     | 25158/49152 [00:13<00:09, 2484.17it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  52%|█████▏    | 25407/49152 [00:13<00:09, 2484.70it/s]�[32m [repeated 24x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  93%|█████████▎| 45954/49152 [00:18<00:00, 9083.17it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  98%|█████████▊| 47955/49152 [00:18<00:00, 12265.89it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])): 100%|██████████| 49152/49152 [00:18<00:00, 2637.36it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):   0%|          | 0/49152 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  93%|█████████▎| 45953/49152 [00:18<00:00, 9008.17it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  97%|█████████▋| 47823/49152 [00:18<00:00, 11803.95it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  12%|█▏        | 5765/49152 [00:02<00:18, 2324.80it/s]�[32m [repeated 625x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  15%|█▌        | 7471/49152 [00:03<00:17, 2418.47it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  16%|█▌        | 7718/49152 [00:03<00:17, 2431.47it/s]�[32m [repeated 20x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  97%|█████████▋| 47809/49152 [00:18<00:00, 11902.44it/s]�[32m [repeated 28x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):   0%|          | 0/49152 [00:00<?, ?it/s]�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])):  97%|█████████▋| 47809/49152 [00:18<00:00, 11970.86it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])): 100%|██████████| 49152/49152 [00:18<00:00, 2599.39it/s] �[32m [repeated 7x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  40%|████      | 19726/49152 [00:07<00:08, 3311.97it/s]�[32m [repeated 698x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70522)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  42%|████▏     | 20743/49152 [00:07<00:08, 3413.91it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  43%|████▎     | 21098/49152 [00:07<00:08, 3452.94it/s]�[32m [repeated 49x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([256, 7168])): 100%|██████████| 49152/49152 [00:18<00:00, 2595.57it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  94%|█████████▍| 46274/49152 [00:11<00:00, 13536.16it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])): 100%|█████████▉| 49089/49152 [00:11<00:00, 17889.35it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])): 100%|██████████| 49152/49152 [00:11<00:00, 4132.95it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):   0%|          | 0/2064 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45580, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  25%|██▍       | 513/2064 [00:00<00:00, 4531.99it/s]�[32m [repeated 642x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  58%|█████▊    | 28708/49152 [00:09<00:04, 4509.03it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  59%|█████▉    | 29189/49152 [00:09<00:04, 4599.13it/s]�[32m [repeated 39x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  93%|█████████▎| 45865/49152 [00:11<00:00, 12764.93it/s]�[32m [repeated 21x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  99%|█████████▉| 48804/49152 [00:12<00:00, 17723.33it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])): 100%|██████████| 49152/49152 [00:12<00:00, 4088.91it/s] �[32m [repeated 8x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):   0%|          | 0/2064 [00:00<?, ?it/s]�[32m [repeated 14x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  25%|██▍       | 513/2064 [00:00<00:00, 4499.05it/s]�[32m [repeated 54x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  35%|███▍      | 717/2064 [00:11<00:00, 4563.92it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  35%|███▍      | 718/2064 [00:11<00:26, 51.27it/s]  �[32m [repeated 4x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  93%|█████████▎| 45680/49152 [00:11<00:00, 12359.64it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])):  99%|█████████▉| 48685/49152 [00:12<00:00, 17604.10it/s]
DeepGemm(fp8_gemm_nt) warmup (W=torch.Size([7168, 128])): 100%|██████████| 49152/49152 [00:12<00:00, 4081.16it/s] 
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):   0%|          | 0/2064 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  34%|███▍      | 712/2064 [00:10<00:26, 51.68it/s]�[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  38%|███▊      | 775/2064 [00:14<00:00, 4497.08it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  38%|███▊      | 776/2064 [00:14<00:29, 44.10it/s]  �[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  49%|████▉     | 1014/2064 [00:24<00:35, 29.25it/s]�[32m [repeated 17x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  43%|████▎     | 884/2064 [00:19<00:00, 4535.75it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  43%|████▎     | 885/2064 [00:19<00:30, 38.07it/s]  �[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  52%|█████▏    | 1077/2064 [00:27<00:31, 30.94it/s]�[32m [repeated 18x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  62%|██████▏   | 1281/2064 [00:35<00:27, 28.23it/s]�[32m [repeated 54x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  63%|██████▎   | 1304/2064 [00:36<00:26, 28.54it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  63%|██████▎   | 1307/2064 [00:36<00:26, 28.61it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  66%|██████▌   | 1360/2064 [00:38<00:24, 28.99it/s]�[32m [repeated 175x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  65%|██████▌   | 1348/2064 [00:38<00:24, 29.29it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  65%|██████▌   | 1351/2064 [00:38<00:24, 29.36it/s]�[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  78%|███████▊  | 1610/2064 [00:45<00:12, 37.61it/s]�[32m [repeated 612x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  78%|███████▊  | 1614/2064 [00:45<00:11, 37.54it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  78%|███████▊  | 1618/2064 [00:45<00:11, 37.67it/s]�[32m [repeated 3x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  83%|████████▎ | 1715/2064 [00:48<00:08, 41.71it/s]�[32m [repeated 704x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  91%|█████████▏| 1886/2064 [00:51<00:03, 52.06it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  92%|█████████▏| 1892/2064 [00:51<00:03, 52.49it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  92%|█████████▏| 1898/2064 [00:51<00:03, 52.93it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  92%|█████████▏| 1904/2064 [00:51<00:02, 53.39it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  93%|█████████▎| 1910/2064 [00:51<00:02, 53.85it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  93%|█████████▎| 1916/2064 [00:52<00:02, 54.32it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  93%|█████████▎| 1922/2064 [00:52<00:02, 54.80it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  93%|█████████▎| 1928/2064 [00:52<00:02, 55.30it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  94%|█████████▎| 1934/2064 [00:52<00:02, 55.79it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  94%|█████████▍| 1940/2064 [00:52<00:02, 56.31it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  94%|█████████▍| 1946/2064 [00:52<00:02, 56.82it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  88%|████████▊ | 1811/2064 [00:50<00:05, 46.85it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  88%|████████▊ | 1816/2064 [00:50<00:05, 47.15it/s]�[32m [repeated 9x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  90%|████████▉ | 1853/2064 [00:51<00:04, 49.20it/s]�[32m [repeated 539x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  93%|█████████▎| 1913/2064 [00:52<00:02, 53.36it/s]�[32m [repeated 142x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  97%|█████████▋| 2008/2064 [00:57<00:03, 17.64it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])): 100%|██████████| 2064/2064 [00:57<00:00, 35.59it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):   0%|          | 0/2064 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  87%|████████▋ | 1789/2064 [00:50<00:06, 45.19it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  87%|████████▋ | 1794/2064 [00:50<00:05, 45.47it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  25%|██▍       | 515/2064 [00:00<00:00, 4392.86it/s]�[32m [repeated 41x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  95%|█████████▍| 1955/2064 [00:58<00:29,  3.64it/s]�[32m [repeated 57x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  97%|█████████▋| 2008/2064 [00:58<00:03, 16.71it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])): 100%|██████████| 2064/2064 [00:58<00:00, 35.33it/s]�[32m [repeated 5x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):   0%|          | 0/2064 [00:00<?, ?it/s]�[32m [repeated 13x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  46%|████▋     | 959/2064 [00:11<00:15, 70.62it/s]  �[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  95%|█████████▍| 1952/2064 [00:58<00:31,  3.59it/s]�[32m [repeated 4x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])):  97%|█████████▋| 2008/2064 [00:58<00:03, 17.36it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 4096, 7168])): 100%|██████████| 2064/2064 [00:58<00:00, 35.02it/s]�[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):   0%|          | 0/2064 [00:00<?, ?it/s]�[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  45%|████▍     | 920/2064 [00:10<00:00, 4399.07it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  45%|████▍     | 921/2064 [00:10<00:15, 72.29it/s]  
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70186)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  54%|█████▍    | 1110/2064 [00:14<00:15, 60.71it/s]�[32m [repeated 17x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  45%|████▌     | 930/2064 [00:10<00:00, 4406.42it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  45%|████▌     | 931/2064 [00:10<00:15, 71.61it/s]  �[32m [repeated 11x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45579, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  66%|██████▌   | 1361/2064 [00:19<00:12, 57.58it/s]�[32m [repeated 84x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45581, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  75%|███████▍  | 1540/2064 [00:22<00:07, 67.66it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  75%|███████▍  | 1547/2064 [00:22<00:07, 68.15it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  91%|█████████▏| 1888/2064 [00:26<00:01, 100.64it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  92%|█████████▏| 1899/2064 [00:26<00:01, 102.15it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  93%|█████████▎| 1910/2064 [00:26<00:01, 103.70it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  93%|█████████▎| 1921/2064 [00:26<00:01, 105.31it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  94%|█████████▎| 1933/2064 [00:26<00:01, 107.12it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  94%|█████████▍| 1945/2064 [00:26<00:01, 108.97it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  95%|█████████▍| 1957/2064 [00:26<00:00, 110.89it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  95%|█████████▌| 1969/2064 [00:26<00:00, 112.89it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  81%|████████▏ | 1677/2064 [00:24<00:04, 77.56it/s]�[32m [repeated 554x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  96%|█████████▌| 1982/2064 [00:27<00:00, 115.12it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  92%|█████████▏| 1893/2064 [00:26<00:01, 100.51it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  92%|█████████▏| 1904/2064 [00:26<00:01, 102.03it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m 2025-10-10 20:28:43,579 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69558)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  81%|████████  | 1673/2064 [00:24<00:05, 77.33it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  81%|████████▏ | 1681/2064 [00:24<00:04, 78.01it/s]�[32m [repeated 4x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  96%|█████████▌| 1978/2064 [00:27<00:00, 113.39it/s]�[32m [repeated 102x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  89%|████████▉ | 1835/2064 [00:26<00:02, 92.49it/s]�[32m [repeated 345x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45609, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  96%|█████████▋| 1989/2064 [00:29<00:05, 14.53it/s] 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])): 100%|██████████| 2064/2064 [00:29<00:00, 69.29it/s]�[32m [repeated 9x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 2025-10-10 20:28:46,859 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...�[32m [repeated 12x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69559)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  90%|████████▉ | 1849/2064 [00:26<00:02, 94.32it/s]
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  90%|█████████ | 1859/2064 [00:26<00:02, 95.55it/s]�[32m [repeated 2x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m 2025-10-10 20:28:59,958 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])): 100%|██████████| 2064/2064 [00:30<00:00, 68.73it/s]�[32m [repeated 45x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 
DeepGemm(m_grouped_fp8_gemm_nt_contiguous) warmup (W=torch.Size([16, 7168, 2048])):  91%|█████████ | 1876/2064 [00:26<00:01, 97.55it/s]�[32m [repeated 7x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45583, ip=172.16.0.48)�[0m 2025-10-10 20:28:50,084 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...�[32m [repeated 3x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/7 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  14%|█▍        | 1/7 [00:00<00:02,  2.05it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  29%|██▊       | 2/7 [00:00<00:01,  3.05it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 3/7 [00:00<00:01,  3.63it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  57%|█████▋    | 4/7 [00:01<00:00,  4.00it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████▏  | 5/7 [00:01<00:00,  4.23it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  86%|████████▌ | 6/7 [00:01<00:00,  4.34it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:02<00:00,  3.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:02<00:00,  3.34it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=69924)�[0m 
Capturing CUDA graphs (decode, FULL):   0%|          | 0/6 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   0%|          | 0/6 [00:00<?, ?it/s]
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45577, ip=172.16.0.48)�[0m WARNING 10-10 20:29:00 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=16,N=2048,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[128,128].json']
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45578, ip=172.16.0.48)�[0m INFO 10-10 20:21:29 [gpu_worker.py:298] Available KV cache memory: 26.98 GiB�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] EngineCore failed to start.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     engine_core = EngineCoreProc(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 498, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     super().__init__(vllm_config, executor_class, log_stats,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 92, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._initialize_kv_caches(vllm_config)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 207, in _initialize_kv_caches
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self.model_executor.initialize_from_config(kv_cache_configs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self.collective_rpc("compile_or_warm_up_model")
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/executor_base.py", line 312, in collective_rpc
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return self._run_workers(method, *args, **(kwargs or {}))
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/ray_distributed_executor.py", line 505, in _run_workers
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     ray_worker_outputs = ray.get(ray_worker_outputs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return fn(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2882, in get
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 968, in get_objects
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     raise value.as_instanceof_cause()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] ray.exceptions.RayTaskError(AssertionError): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45582, ip=172.16.0.48, actor_id=99c37bbc4af57d1e9c3c27ea01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fe35d6c86b0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ERROR 10-10 20:29:02 [core.py:708] AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m Process EngineCore_DP0:
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self.run()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._target(*self._args, **self._kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 712, in run_engine_core
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     engine_core = EngineCoreProc(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 498, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     super().__init__(vllm_config, executor_class, log_stats,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 92, in __init__
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._initialize_kv_caches(vllm_config)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core.py", line 207, in _initialize_kv_caches
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self.model_executor.initialize_from_config(kv_cache_configs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self.collective_rpc("compile_or_warm_up_model")
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/executor_base.py", line 312, in collective_rpc
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return self._run_workers(method, *args, **(kwargs or {}))
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/executor/ray_distributed_executor.py", line 505, in _run_workers
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     ray_worker_outputs = ray.get(ray_worker_outputs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return fn(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2882, in get
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 968, in get_objects
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise value.as_instanceof_cause()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m ray.exceptions.RayTaskError(AssertionError): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45582, ip=172.16.0.48, actor_id=99c37bbc4af57d1e9c3c27ea01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fe35d6c86b0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:02,994	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45609, ip=172.16.0.48, actor_id=3dda0e47c6eb9fa1586824d001000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f785a4d87a0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m INFO 10-10 20:29:02 [ray_distributed_executor.py:122] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,006	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45583, ip=172.16.0.48, actor_id=5f7dfbcf8012d45ed1403c3401000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f30d18d0d70>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,007	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45581, ip=172.16.0.48, actor_id=d12f607e858ae0a9d6f00b5e01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fd496444ad0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,008	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=70644, ip=172.16.0.44, actor_id=47d9565e351801ab78c4c52d01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f41bd09cd10>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,017	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=70306, ip=172.16.0.44, actor_id=73b8b6cbbdff15c15c4b81a201000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f0726d08d10>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,017	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=70522, ip=172.16.0.44, actor_id=22066639175f65a060f99c1301000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fdd5d42d070>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,080	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=69558, ip=172.16.0.44, actor_id=e8fc606e386bb4b09784e66f01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fc92f670f20>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,081	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=70186, ip=172.16.0.44, actor_id=4b7334dd210dd12dfad2767e01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f44cb8413a0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,109	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45577, ip=172.16.0.48, actor_id=fb7bd612e8b7981ceab8538601000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f2ccb280b00>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,117	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45579, ip=172.16.0.48, actor_id=dc8445d6522eddb9afe6543f01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fc872ed4c50>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,120	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45578, ip=172.16.0.48, actor_id=62486acb9631aaefbee597f401000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f75cf1a0aa0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,134	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=69924, ip=172.16.0.44, actor_id=3a80f650ef55d258bf06800901000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f77a8204ef0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,136	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=45580, ip=172.16.0.48, actor_id=cea994c9653baa40e939d9fc01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7ffb072bcda0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,140	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=69559, ip=172.16.0.44, actor_id=1ca096b5ea9599f63bc7b8ea01000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fbc4880cbc0>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m 2025-10-10 20:29:03,231	ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=70325, ip=172.16.0.44, actor_id=ce14c3e2ce19d9d5c8347d6801000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f7f06b68f80>)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 276, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     raise e
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m 2025-10-10 20:28:59,958 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275] Error executing method 'compile_or_warm_up_model'. This might cause deadlock in distributed execution.
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     return run_method(self, method, args, kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     cuda_graph_memory_bytes = self.model_runner.capture_model()
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     self._capture_cudagraphs(
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     self._dummy_run(num_tokens,
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     return func(*args, **kwargs)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     .build_for_cudagraph_capture(common_attn_metadata)
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]     assert m.max_query_len <= self.reorder_batch_threshold  # decode only
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=45582, ip=172.16.0.48)�[0m ERROR 10-10 20:29:02 [worker_base.py:275] AssertionError
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70644)�[0m WARNING 10-10 20:29:00 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=16,N=2048,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[128,128].json']�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275] Error executing method 'compile_or_warm_up_model'. This might cause deadlock in distributed execution.�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275] Traceback (most recent call last):�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 267, in execute_method�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     return run_method(self, method, args, kwargs)�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 3122, in run_method�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     return func(*args, **kwargs)�[32m [repeated 30x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^�[32m [repeated 30x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     cuda_graph_memory_bytes = self.model_runner.capture_model()�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3463, in capture_model�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     self._capture_cudagraphs(�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3523, in _capture_cudagraphs�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     self._dummy_run(num_tokens,�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3080, in _dummy_run�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     .build_for_cudagraph_capture(common_attn_metadata)�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/common.py", line 652, in build_for_cudagraph_capture�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]     assert m.max_query_len <= self.reorder_batch_threshold  # decode only�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[32m [repeated 15x across cluster]�[0m
�[1;36m(EngineCore_DP0 pid=69386)�[0;0m �[36m(RayWorkerWrapper pid=70325)�[0m ERROR 10-10 20:29:03 [worker_base.py:275] AssertionError�[32m [repeated 15x across cluster]�[0m
�[1;36m(APIServer pid=69250)�[0;0m Traceback (most recent call last):
�[1;36m(APIServer pid=69250)�[0;0m   File "<frozen runpy>", line 198, in _run_module_as_main
�[1;36m(APIServer pid=69250)�[0;0m   File "<frozen runpy>", line 88, in _run_code
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/cli/main.py", line 60, in <module>
�[1;36m(APIServer pid=69250)�[0;0m     main()
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/cli/main.py", line 54, in main
�[1;36m(APIServer pid=69250)�[0;0m     args.dispatch_function(args)
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/cli/serve.py", line 57, in cmd
�[1;36m(APIServer pid=69250)�[0;0m     uvloop.run(run_server(args))
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
�[1;36m(APIServer pid=69250)�[0;0m     return __asyncio.run(
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
�[1;36m(APIServer pid=69250)�[0;0m     return runner.run(main)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
�[1;36m(APIServer pid=69250)�[0;0m     return self._loop.run_until_complete(task)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
�[1;36m(APIServer pid=69250)�[0;0m     return await main
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/openai/api_server.py", line 1884, in run_server
�[1;36m(APIServer pid=69250)�[0;0m     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker
�[1;36m(APIServer pid=69250)�[0;0m     async with build_async_engine_client(
�[1;36m(APIServer pid=69250)�[0;0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
�[1;36m(APIServer pid=69250)�[0;0m     return await anext(self.gen)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client
�[1;36m(APIServer pid=69250)�[0;0m     async with build_async_engine_client_from_engine_args(
�[1;36m(APIServer pid=69250)�[0;0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
�[1;36m(APIServer pid=69250)�[0;0m     return await anext(self.gen)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/entrypoints/openai/api_server.py", line 225, in build_async_engine_client_from_engine_args
�[1;36m(APIServer pid=69250)�[0;0m     async_llm = AsyncLLM.from_vllm_config(
�[1;36m(APIServer pid=69250)�[0;0m                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/utils/__init__.py", line 1572, in inner
�[1;36m(APIServer pid=69250)�[0;0m     return fn(*args, **kwargs)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config
�[1;36m(APIServer pid=69250)�[0;0m     return cls(
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/async_llm.py", line 134, in __init__
�[1;36m(APIServer pid=69250)�[0;0m     self.engine_core = EngineCoreClient.make_async_mp_client(
�[1;36m(APIServer pid=69250)�[0;0m                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
�[1;36m(APIServer pid=69250)�[0;0m     return AsyncMPClient(*client_args)
�[1;36m(APIServer pid=69250)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core_client.py", line 769, in __init__
�[1;36m(APIServer pid=69250)�[0;0m     super().__init__(
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/core_client.py", line 448, in __init__
�[1;36m(APIServer pid=69250)�[0;0m     with launch_core_engines(vllm_config, executor_class,
�[1;36m(APIServer pid=69250)�[0;0m          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(APIServer pid=69250)�[0;0m   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
�[1;36m(APIServer pid=69250)�[0;0m     next(self.gen)
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/utils.py", line 732, in launch_core_engines
�[1;36m(APIServer pid=69250)�[0;0m     wait_for_engine_startup(
�[1;36m(APIServer pid=69250)�[0;0m   File "/cfs_zhongwei/qiangqwang/vllm_project/vllm/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup
�[1;36m(APIServer pid=69250)�[0;0m     raise RuntimeError("Engine core initialization failed. "
�[1;36m(APIServer pid=69250)�[0;0m RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

[benchislett]

This should be fixed when #26541 is merged.

[benchislett]

@WangQiangItachi please test again with latest main

[WangQiangItachi]

@WangQiangItachi please test again with latest main

Thanks, the MTP-related issue has been resolved, but the speed of DeepSeek v3.1 is much slower than v0.10.2. Is this a known issue?

[benchislett]

@WangQiangItachi that is not expected. Is it also slower without MTP enabled? Are you able to collect some nsys profiling data to identify where the regression is coming from?

[WangQiangItachi]

@WangQiangItachi that is not expected. Is it also slower without MTP enabled? Are you able to collect some nsys profiling data to identify where the regression is coming from?

I feel it’s not related to MTP. The nsys show that the newly introduced sm90_fp8_gemm_1d2d_impl operator takes up the majority of the time.