[Bug]: support the Qwen3-VL-235B-A22B model？

[gerayking]

Your current environment

The output of python collect_env.py

text ============================== System Info ============================== OS : Server 4.2 (x86_64) GCC version : (Compiler 12.3.1.3) 12.3.1 20230912 (OS 12.3.1.3-3) Clang version : 17.0.6 ( 17.0.6-8.tl4.ap.2) CMake version : Could not collect Libc version : glibc-2.38

==============================
PyTorch Info

PyTorch version : 2.8.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.12.11 | packaged by Anaconda, Inc. | (main, Jun 5 2025, 13:09:17) [GCC 11.2.0] (64-bit runtime)
Python platform : Linux-5.4.241-1-tlinux4-0017.7-x86_64-with-glibc2.38

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 12.4.99
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version : 535.161.08
cuDNN version : Probably one of the following:
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_engines_precompiled.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_engines_runtime_compiled.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_graph.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_heuristic.so.9.5.1
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops.so.9.5.1
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
Model name: AMD EPYC 9K84 96-Core Processor
BIOS Model name: AMD EPYC 9K84 96-Core Processor Unknown CPU @ 2.6GHz
BIOS CPU family: 107
CPU family: 25
Model: 17
Thread(s) per core: 2
Core(s) per socket: 96
Socket(s): 2
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 142%
CPU max MHz: 2600.0000
CPU min MHz: 1500.0000
BogoMIPS: 5200.30
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization: AMD-V
L1d cache: 6 MiB (192 instances)
L1i cache: 6 MiB (192 instances)
L2 cache: 192 MiB (192 instances)
L3 cache: 768 MiB (24 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-95,192-287
NUMA node1 CPU(s): 96-191,288-383
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

==============================
Versions of relevant libraries

[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.0.dev0
[pip3] triton==3.4.0
[conda] numpy 2.2.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.3 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] pyzmq 27.1.0 pypi_0 pypi
[conda] torch 2.8.0 pypi_0 pypi
[conda] torchaudio 2.8.0 pypi_0 pypi
[conda] torchvision 0.23.0 pypi_0 pypi
[conda] transformers 4.57.0.dev0 pypi_0 pypi
[conda] triton 3.4.0 pypi_0 pypi

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.11.0rc2.dev70+gb1068903f (git sha: b106890)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16 NIC17 NIC18 NIC19 NIC20 NIC21 NIC22 NIC23 NIC24 NIC25 NIC26 NIC27 NIC28 NIC29 NIC30 NIC31 NIC32 NIC33 NIC34 NIC35 NIC36 NIC37 NIC38 NIC39 NIC40 NIC41 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PHB PIX SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX PHB SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE PIX NODE 96-191,288-383 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE PIX NODE NODE 96-191,288-383 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PHB NODE NODE PIX 96-191,288-383 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PIX NODE NODE PHB 96-191,288-383 1 N/A
NIC0 SYS SYS SYS SYS NODE NODE NODE NODE X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC1 SYS SYS SYS SYS NODE NODE NODE NODE PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC2 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC3 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC4 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC5 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC6 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC7 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC8 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC9 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC10 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC11 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC12 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC13 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC14 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC15 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC16 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC17 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC18 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC19 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC20 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC21 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC22 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC23 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC24 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC25 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC26 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC27 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC28 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC29 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC30 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC31 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC32 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYS SYS NODE NODE NODE NODE
NIC33 SYS SYS SYS SYS NODE NODE NODE NODE PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYS SYS NODE NODE NODE NODE
NIC34 PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS
NIC35 NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS
NIC36 NODE PHB PIX NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X PHB SYS SYS SYS SYS
NIC37 NODE PIX PHB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PHB X SYS SYS SYS SYS
NIC38 SYS SYS SYS SYS NODE NODE PHB PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS X NODE NODE PHB
NIC39 SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE X NODE NODE
NIC40 SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE
NIC41 SYS SYS SYS SYS NODE NODE PIX PHB NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS PHB NODE NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: NIC41 CPU Affinity NUMA Affinity GPU NUMA ID
NIC1: MA ID
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
NIC14: mlx5_14
NIC15: mlx5_15
NIC16: mlx5_16
NIC17: mlx5_17
NIC18: mlx5_18
NIC19: mlx5_19
NIC20: mlx5_20
NIC21: mlx5_21
NIC22: mlx5_22
NIC23: mlx5_23
NIC24: mlx5_24
NIC25: mlx5_25
NIC26: mlx5_26
NIC27: mlx5_27
NIC28: mlx5_28
NIC29: mlx5_29
NIC30: mlx5_30
NIC31: mlx5_31
NIC32: mlx5_32
NIC33: mlx5_33
NIC34: mlx5_bond_1
NIC35: mlx5_bond_2
NIC36: mlx5_bond_3
NIC37: mlx5_bond_4
NIC38: mlx5_bond_5
NIC39: mlx5_bond_6
NIC40: mlx5_bond_7
NIC41: mlx5_bond_8

==============================
Environment Variables

NVIDIA_VISIBLE_DEVICES=GPU-fda0e613-4dc5-f03d-0792-0512ead9097a,GPU-253be886-3b58-d189-5fda-dfd022443f7a,GPU-14737678-da38-155c-9c2e-09b3df9b02cd,GPU-d252a9b4-0791-cfe7-4508-48ac4cdda503,GPU-4a0559e1-c4a4-86d6-673e-3cd12f123a2c,GPU-b108e04e-985b-ac55-3b5d-f6ddaee1bb1c,GPU-4619eae8-17a4-a148-15ab-4a48eb322a96,GPU-7cb521e9-d609-948d-6053-adf7b7dc9d64
NCCL_SOCKET_IFNAME=bond1
NCCL_NET_GDR_LEVEL=2
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NCCL_DEBUG=INFO
NCCL_NSOCKS_PERTHREAD=4
NCCL_IB_GID_INDEX=3
NCCL_SOCKET_NTHREADS=4
NCCL_IB_SL=3
LD_LIBRARY_PATH=/usr/local/cuda/lib64::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/extras/CUPTI/lib64/:/usr/local/cuda-12.4/compat/:/usr/local/cuda/compat/
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I attempted to run the Qwen3-VL model on vLLM with the following command: vllm serve /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export --served-model-name Qwen3-VL-235B-A22B --tensor-parallel-size 4 --pipeline-parallel-size 2 --limit-mm-per-prompt '{ "image":8, "video":1}'

However, I encountered an error.

(Worker_PP0_TP2 pid=72464) INFO 09-24 20:54:47 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP0_TP0 pid=72462) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP1_TP0 pid=72466) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP1_TP2 pid=72468) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP1_TP1 pid=72467) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP1_TP3 pid=72469) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP0_TP1 pid=72463) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP0_TP3 pid=72465) INFO 09-24 20:54:48 [gpu_model_runner.py:2539] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_PP0_TP0 pid=72462) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP1_TP2 pid=72468) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP1_TP3 pid=72469) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP1_TP0 pid=72466) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP0_TP0 pid=72462) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP1_TP2 pid=72468) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP1_TP3 pid=72469) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP1_TP0 pid=72466) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP0_TP2 pid=72464) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP0_TP2 pid=72464) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP0_TP1 pid=72463) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP1_TP1 pid=72467) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP0_TP1 pid=72463) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards:   0% Completed | 0/96 [00:00<?, ?it/s]
(Worker_PP1_TP1 pid=72467) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_PP0_TP3 pid=72465) INFO 09-24 20:54:48 [gpu_model_runner.py:2571] Loading model from scratch...
(Worker_PP0_TP3 pid=72465) INFO 09-24 20:54:48 [cuda.py:347] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards:   1% Completed | 1/96 [00:00<00:19,  4.97it/s]
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597] WorkerProc failed to start.
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597] Traceback (most recent call last):
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     worker = WorkerProc(*args, **kwargs)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/executor/multiproc_executor.py", line 437, in __init__
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.worker.load_model()
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/worker/gpu_worker.py", line 221, in load_model
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/worker/gpu_model_runner.py", line 2572, in load_model
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.model = model_loader.load_model(
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.load_weights(model, model_config)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/model_loader/default_loader.py", line 264, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_weights = model.load_weights(
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                      ^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl.py", line 1511, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     yield from self._load_module(prefix,
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_params = module_load_weights(weights)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_moe.py", line 699, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     return loader.load_weights(weights)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     yield from self._load_module(prefix,
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_params = module_load_weights(weights)
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl_moe.py", line 229, in load_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     success = self.load_fused_expert_weights(
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl_moe.py", line 126, in load_fused_expert_weights
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     param = params_dict[name]
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597]             ~~~~~~~~~~~^^^^^^
(Worker_PP1_TP3 pid=72469) ERROR 09-24 20:54:49 [multiproc_executor.py:597] KeyError: 'layers.0.mlp.experts.w2_weight'
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597] WorkerProc failed to start.
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597] Traceback (most recent call last):
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     worker = WorkerProc(*args, **kwargs)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/executor/multiproc_executor.py", line 437, in __init__
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.worker.load_model()
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/worker/gpu_worker.py", line 221, in load_model
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/v1/worker/gpu_model_runner.py", line 2572, in load_model
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.model = model_loader.load_model(
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     self.load_weights(model, model_config)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/model_loader/default_loader.py", line 264, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_weights = model.load_weights(
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                      ^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl.py", line 1511, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     yield from self._load_module(prefix,
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_params = module_load_weights(weights)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_moe.py", line 699, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     return loader.load_weights(weights)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     yield from self._load_module(prefix,
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     loaded_params = module_load_weights(weights)
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl_moe.py", line 229, in load_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     success = self.load_fused_expert_weights(
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]   File "/mnt/cephfs/user_xuanweifu/vllm/vllm/model_executor/models/qwen3_vl_moe.py", line 126, in load_fused_expert_weights
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]     param = params_dict[name]
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597]             ~~~~~~~~~~~^^^^^^
(Worker_PP1_TP2 pid=72468) ERROR 09-24 20:54:49 [multiproc_executor.py:597] KeyError: 'layers.0.mlp.experts.w2_weight'

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ywang96]

Looks like there's something wrong with pipeline parallelism for this model. Can you try with TP=8?

There's an issue with video profiling we're trying to fix, so I recommend you disable it for now until we fix it on main. Thanks!

vllm serve /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export --served-model-name Qwen3-VL-235B-A22B --tensor-parallel-size 8 --limit-mm-per-prompt.video 0

[Buhua-Liu]

I used 8 80G H100, but got OOM. What is the memory requirement?

[gerayking]

Looks like there's something wrong with pipeline parallelism for this model. Can you try with TP=8?

There's an issue with video profiling we're trying to fix, so I recommend you disable it for now until we fix it on main. Thanks!

vllm serve /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export --served-model-name Qwen3-VL-235B-A22B --tensor-parallel-size 8 --limit-mm-per-prompt.video 0

thanks, I used vllm serve /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export --served-model-name Qwen3-VL-235B-A22B --tensor-parallel-size 8 --limit-mm-per-prompt.video 0 run but I meet OOM too. my device is 8*H20 96G.

[ywang96]

Hmmm - it's interesting because the command I shared can launch on A100. Could you guys share the exact OOM logs from H100 and H20? Thanks! @gerayking @Buhua-Liu

It may also help if you can add --max-num-batched-tokens 2048 and/or --max-num-seqs 128

[gerayking]

Hmmm - it's interesting because the command I shared can launch on A100. Could you guys share the exact OOM logs from H100 and H20? Thanks! @gerayking @Buhua-Liu

It may also help if you can add --max-num-batched-tokens 2048 and/or --max-num-seqs 128

Thanks for your help and your attention, command is vllm serve /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export --served-model-name Qwen3-VL-235B-A22B --tensor-parallel-size 8 --limit-mm-per-prompt.video 0 --max-num-batched-tokens 2048 --max-num-seqs 128

output log

INFO 09-25 14:42:23 [parallel_state.py:1165] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
INFO 09-25 14:42:23 [parallel_state.py:1165] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
WARNING 09-25 14:42:23 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:23 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-25 14:42:24 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(Worker_TP1 pid=122161) INFO 09-25 14:42:25 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP3 pid=122163) INFO 09-25 14:42:25 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP0 pid=122160) INFO 09-25 14:42:25 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP1 pid=122161) INFO 09-25 14:42:25 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP1 pid=122161) INFO 09-25 14:42:25 [transformers.py:439] Using Transformers backend.
(Worker_TP1 pid=122161) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP3 pid=122163) INFO 09-25 14:42:25 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP3 pid=122163) INFO 09-25 14:42:25 [transformers.py:439] Using Transformers backend.
(Worker_TP3 pid=122163) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP7 pid=122167) INFO 09-25 14:42:26 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP4 pid=122164) INFO 09-25 14:42:26 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP6 pid=122166) INFO 09-25 14:42:26 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP5 pid=122165) INFO 09-25 14:42:26 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP2 pid=122162) INFO 09-25 14:42:26 [gpu_model_runner.py:2338] Starting to load model /home/qspace/base_model_Qwen3-VL-235B-A22B-Instruct_export...
(Worker_TP1 pid=122161) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP0 pid=122160) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP0 pid=122160) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP0 pid=122160) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP3 pid=122163) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP4 pid=122164) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP4 pid=122164) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP4 pid=122164) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP7 pid=122167) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP7 pid=122167) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP7 pid=122167) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP6 pid=122166) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP6 pid=122166) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP6 pid=122166) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP2 pid=122162) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP2 pid=122162) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP2 pid=122162) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP5 pid=122165) INFO 09-25 14:42:26 [gpu_model_runner.py:2370] Loading model from scratch...
(Worker_TP5 pid=122165) INFO 09-25 14:42:26 [transformers.py:439] Using Transformers backend.
(Worker_TP5 pid=122165) `torch_dtype` is deprecated! Use `dtype` instead!
(Worker_TP0 pid=122160) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP4 pid=122164) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP7 pid=122167) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP6 pid=122166) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP2 pid=122162) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP5 pid=122165) INFO 09-25 14:42:26 [cuda.py:362] Using Flash Attention backend on V1 engine.
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585] WorkerProc failed to start.
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585] Traceback (most recent call last):
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 559, in worker_main
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     worker = WorkerProc(*args, **kwargs)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 427, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.worker.load_model()
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 213, in load_model
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2371, in load_model
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.model = model_loader.load_model(
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 45, in load_model
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     model = initialize_model(vllm_config=vllm_config,
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 64, in initialize_model
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     return model_class(vllm_config=vllm_config, prefix=prefix)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 199, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 792, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     super().__init__(vllm_config=vllm_config, prefix=prefix)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 199, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 707, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     super().__init__(vllm_config=vllm_config, prefix=prefix)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 485, in __init__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.init_parameters(self.model)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 648, in init_parameters
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.init_parameters(child)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 648, in init_parameters
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.init_parameters(child)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 648, in init_parameters
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     self.init_parameters(child)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   [Previous line repeated 2 more times]
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/vllm/model_executor/models/transformers.py", line 643, in init_parameters
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     torch.empty_like(param.data,
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/torch/utils/_device.py", line 103, in __torch_function__
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     return func(*args, **kwargs)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/torch/_prims_common/wrappers.py", line 309, in _fn
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     result = fn(*args, **kwargs)
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]              ^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]   File "/data/miniconda3/envs/qwen3vl-normal/lib/python3.12/site-packages/torch/_refs/__init__.py", line 5055, in empty_like
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]     return torch.empty_permuted(
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=122161) ERROR 09-25 14:42:26 [multiproc_executor.py:585] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 968.00 MiB is free. Process 2147878 has 94.05 GiB memory in use. Of the allocated memory 92.90 GiB is allocated by PyTorch, and 38.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[ywang96]

@gerayking Can you verify you're using the vllm nightly wheel? If so this model should not be using transformers backend as it says in your log.

You can install the nightly with
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

[KTA-LAB]

Hello, I’m currently trying to use Qwen3-VL and facing an issue.

I successfully deployed the model using the example code provided in the Qwen3-VL GitHub repository. When I tested the model directly with transformers, it worked correctly. However, when I used the provided OpenAI-compatible example code, the model seems unable to read the image at all and produces irrelevant responses.

Here’s the exact code I ran:

import time
from openai import OpenAI

client = OpenAI(
api_key="EMPTY",
base_url="http://127.0.0.1:22002/v1",
timeout=3600
)

messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]

start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

When I sent this request, the model just generated the phrase “BONUS BALL” repeatedly until the max token limit was reached. This text has nothing to do with the receipt image I provided.

I’d like to know what might be causing this behavior. Is this a limitation of the current OpenAI-compatible interface, or am I missing something in how the image input should be formatted?

Thanks in advance!

[ericyuegu]

I am seeing the exact same behavior as @KTA-LAB on a B200 node. HF example works, vLLM requests produce a single token repeated until max token limit.

The command I'm running:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tokenizer Qwen/Qwen3-VL-235B-A22B-Instruct \
  --served-model-name Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --enable-expert-parallel \
  --host 0.0.0.0 \
  --port 22002 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.70 \
  --distributed-executor-backend mp

I'm inside the Qwen-provided Docker:
docker run --gpus all --ipc=host --network=host --rm --name qwen3vl -it qwenllm/qwenvl:qwen3vl-cu128 bash

pip list shows:

transformers                      4.57.0.dev0
vllm                              0.11.0rc2.dev117+ge04a1b6b2

I tried disabling FlashInfer to check if FlashAttention2/SPDA/FlexAttention or any other attn kernel might resolve the issue, but it appears that Qwen3-VL doesn't yet support any other backend.

[ywang96]

Thanks @ericyuegu @KTA-LAB for reporting the issues! I think you're describing two different issues here

@KTA-LAB If I understand correctly, the issue is that using qwen_vl_utils to process the image then feed them to vLLM generates the correct result, but the online server interface doesn't?

@ericyuegu Can you share a repro example? Qwen3-VL should be able to support other ViT backend so I'll take a look at that too

[ywang96]

@KTA-LAB btw - I ran the image in your example on 8xA100s served with the following arg
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct -tp 8 --max-model-len 128000 --gpu-memory-utilization 0.98

and got the following response

This image shows a **receipt from Auntie Anne’s**, a popular pretzel and snack shop chain.

Here’s what’s on the receipt:

- **Item Purchased**:  
  **Cinnamon Sugar** (likely a cinnamon sugar pretzel or pretzel snack)  
  Quantity: 1  
  Price: **17,000 IDR** (Indonesian Rupiah)

- **Subtotal**: 17,000 IDR  
- **Grand Total**: 17,000 IDR  
- **Cash Paid**: 20,000 IDR  
- **Change Due**: 3,000 IDR

The receipt is printed on white thermal paper, and the top portion (including store address and receipt number) is blurred or obscured. The logo of Auntie Anne’s — a pretzel with a halo — is visible on the left.

This appears to be a transaction from an Auntie Anne’s location in **Indonesia**, given the currency (IDR).

✅ In short: Someone bought a cinnamon sugar pretzel for 17,000 IDR, paid with 20,000 IDR, and received 3,000 IDR in change.

[KTA-LAB]

I haven’t tried offline inference with vLLM before, but I just tested it.

Transformers → works fine with the following code (see below).

vLLM OpenAI-compatible API on B200 → same issue as @ericyuegu: repeats a single token until max token limit when image input is included.

Text-only input → works fine.

Offline inference with vLLM → also problematic with FlashInfer. If I remove FlashInfer, it falls back to FlashAttention2 but it's not working also.

Here’s the transformers code that produced correct results:

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
"Qwen/Qwen3-VL-235B-A22B-Instruct", dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-235B-A22B-Instruct")

messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]

inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

So it looks like the issue is specifically with vLLM + B200, while transformers runs as expected.

Offline inferance code:

-- coding: utf-8 --

import torch
from qwen_vl_utils import process_vision_info
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

import os
os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'

def prepare_inputs_for_vllm(messages, processor):
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# qwen_vl_utils 0.0.14+ reqired
image_inputs, video_inputs, video_kwargs = process_vision_info(messages,
image_patch_size=processor.image_processor.patch_size, return_video_kwargs=True, return_video_metadata=True)
print(f"video_kwargs: {video_kwargs}")

mm_data = {}
if image_inputs is not None:
    mm_data['image'] = image_inputs
if video_inputs is not None:
    mm_data['video'] = video_inputs

return {
    'prompt': text,
    'multi_modal_data': mm_data,
    'mm_processor_kwargs': video_kwargs
}

if name == 'main':
# messages = [
# {
# "role": "user",
# "content": [
# {
# "type": "video",
# "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-VL/space_woaudio.mp4",
# },
# {"type": "text", "text": "这段视频有多长"},
# ],
# }
# ]

messages = [
    {
        "role": "user",
        "content": [
          {
              "type": "image",
              "image": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png",
          },
          {"type": "text", "text": "Read all the text in the image."},
        ],
    }
]

# TODO: change to your own checkpoint path
checkpoint_path = "/data/qwen3-vl"
processor = AutoProcessor.from_pretrained(checkpoint_path)
inputs = [prepare_inputs_for_vllm(message, processor) for message in [messages]]

llm = LLM(
    model=checkpoint_path, trust_remote_code=True, 
  	gpu_memory_utilization=0.70, enforce_eager=False,
    tensor_parallel_size=torch.cuda.device_count(),
    seed=0
)

sampling_params = SamplingParams(
    temperature=0,
    max_tokens=1024,
    top_k=-1,
    stop_token_ids=[],
)

for i, input_ in enumerate(inputs):
    print()
    print('=' * 40)
    print(f"Inputs[{i}]: {input_['prompt']=!r}")
print('\n' + '>' * 40)

outputs = llm.generate(inputs, sampling_params=sampling_params)
for i, output in enumerate(outputs):
    generated_text = output.outputs[0].text
    print()
    print('=' * 40)
    print(f"Generated text: {generated_text!r}")

Offline Inference with flashinfer logs:
(kta) kta@kta-02:~$ python test_qwen3_vl.py
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:38:44 [init.py:216] Automatically detected platform cuda.
video_kwargs: {'do_sample_frames': False}
INFO 09-26 09:39:23 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.7, 'disable_log_stats': True, 'model': '/data/qwen3-vl'}
The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
INFO 09-26 09:39:23 [model.py:540] Resolved architecture: Qwen3VLMoeForConditionalGeneration
torch_dtype is deprecated! Use dtype instead!
INFO 09-26 09:39:23 [model.py:1502] Using max model len 262144
INFO 09-26 09:39:23 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=16384.
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:39:28 [init.py:216] Automatically detected platform cuda.
(EngineCore_DP0 pid=2190849) INFO 09-26 09:39:29 [core.py:644] Waiting for init message from front-end.
(EngineCore_DP0 pid=2190849) INFO 09-26 09:39:29 [core.py:77] Initializing a V1 LLM engine (v0.11.0rc2.dev131+g081b5594a) with config: model='/data/qwen3-vl', speculative_config=None, tokenizer='/data/qwen3-vl', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/data/qwen3-vl, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
(EngineCore_DP0 pid=2190849) WARNING 09-26 09:39:29 [multiproc_executor.py:720] Reducing Torch parallelism from 144 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore_DP0 pid=2190849) INFO 09-26 09:39:29 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3, 4, 5, 6, 7], buffer_handle=(8, 16777216, 10, 'psm_27e35a51'), local_subscribe_addr='ipc:///tmp/94bc4415-38e8-43da-8b81-51f2c1710aab', remote_subscribe_addr=None, remote_addr_ipv6=False)
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:39:33 [init.py:216] Automatically detected platform cuda.
W0926 09:39:44.353000 2190995 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.353000 2190995 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0926 09:39:44.686000 2190990 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.686000 2190990 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
INFO 09-26 09:39:44 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c872afb2'), local_subscribe_addr='ipc:///tmp/52e83378-e4f4-4a95-96f7-9b58931d6431', remote_subscribe_addr=None, remote_addr_ipv6=False)
W0926 09:39:44.744000 2190992 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.744000 2190992 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0926 09:39:44.766000 2190988 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.766000 2190988 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0926 09:39:44.800000 2190993 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.800000 2190993 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
INFO 09-26 09:39:44 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_93a6afce'), local_subscribe_addr='ipc:///tmp/807b6f03-fa63-4fc3-88a0-1d2790426c7a', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:39:44 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_a75511ee'), local_subscribe_addr='ipc:///tmp/07ccc0de-610d-4340-bb06-b933831e977d', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:39:44 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_a550c9b1'), local_subscribe_addr='ipc:///tmp/208234bf-f17f-4175-a2e7-ed8d7ca0c060', remote_subscribe_addr=None, remote_addr_ipv6=False)
W0926 09:39:44.952000 2190989 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.952000 2190989 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
INFO 09-26 09:39:44 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_e63bdf5a'), local_subscribe_addr='ipc:///tmp/427afe7f-82fd-41b6-9c39-2d43820790c5', remote_subscribe_addr=None, remote_addr_ipv6=False)
W0926 09:39:44.963000 2190994 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.963000 2190994 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0926 09:39:44.983000 2190991 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W0926 09:39:44.983000 2190991 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
INFO 09-26 09:39:45 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_2d351490'), local_subscribe_addr='ipc:///tmp/cd00b3b8-a553-4ae9-a26e-84ddeed53f54', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:39:45 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_e15555b7'), local_subscribe_addr='ipc:///tmp/eb35946c-9565-4010-b350-f28c0989d7f0', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:39:45 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_302161cf'), local_subscribe_addr='ipc:///tmp/17f967ed-96bd-4a82-bab5-28a5201754f5', remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:47 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:47 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:47 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:47 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:47 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:47 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:47 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:47 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:47 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:47 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:48 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:48 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:48 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:48 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:48 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:39:48 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:39:52 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_f9a22109'), local_subscribe_addr='ipc:///tmp/e55878c0-591a-4458-b1b4-eefdb9bd8097', remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
INFO 09-26 09:39:52 [parallel_state.py:1201] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
INFO 09-26 09:39:53 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP1 pid=2190989) INFO 09-26 09:39:55 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP4 pid=2190992) INFO 09-26 09:39:55 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP5 pid=2190993) INFO 09-26 09:39:55 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP2 pid=2190990) INFO 09-26 09:39:55 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP0 pid=2190988) INFO 09-26 09:39:55 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP1 pid=2190989) INFO 09-26 09:39:55 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP1 pid=2190989) INFO 09-26 09:39:55 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP4 pid=2190992) INFO 09-26 09:39:55 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP5 pid=2190993) INFO 09-26 09:39:55 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP2 pid=2190990) INFO 09-26 09:39:55 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP4 pid=2190992) INFO 09-26 09:39:55 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP0 pid=2190988) INFO 09-26 09:39:55 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP5 pid=2190993) INFO 09-26 09:39:55 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP2 pid=2190990) INFO 09-26 09:39:55 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP0 pid=2190988) INFO 09-26 09:39:55 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP3 pid=2190991) INFO 09-26 09:39:56 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP7 pid=2190995) INFO 09-26 09:39:56 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP6 pid=2190994) INFO 09-26 09:39:56 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP3 pid=2190991) INFO 09-26 09:39:56 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP7 pid=2190995) INFO 09-26 09:39:56 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP3 pid=2190991) INFO 09-26 09:39:56 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP6 pid=2190994) INFO 09-26 09:39:56 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP7 pid=2190995) INFO 09-26 09:39:56 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
(Worker_TP6 pid=2190994) INFO 09-26 09:39:56 [cuda.py:325] Using FlashInfer backend with HND KV cache layout on V1 engine by default for Blackwell (SM 10.0) GPUs.
Loading safetensors checkpoint shards: 0% Completed | 0/96 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 1% Completed | 1/96 [00:00<00:12, 7.41it/s]
Loading safetensors checkpoint shards: 2% Completed | 2/96 [00:03<02:52, 1.84s/it]
Loading safetensors checkpoint shards: 3% Completed | 3/96 [00:06<03:37, 2.34s/it]
Loading safetensors checkpoint shards: 4% Completed | 4/96 [00:09<03:58, 2.59s/it]
Loading safetensors checkpoint shards: 5% Completed | 5/96 [00:11<04:05, 2.70s/it]
Loading safetensors checkpoint shards: 6% Completed | 6/96 [00:14<03:58, 2.66s/it]
Loading safetensors checkpoint shards: 7% Completed | 7/96 [00:16<03:39, 2.47s/it]
Loading safetensors checkpoint shards: 8% Completed | 8/96 [00:18<03:26, 2.35s/it]
Loading safetensors checkpoint shards: 9% Completed | 9/96 [00:20<03:18, 2.28s/it]
Loading safetensors checkpoint shards: 10% Completed | 10/96 [00:22<03:11, 2.23s/it]
Loading safetensors checkpoint shards: 11% Completed | 11/96 [00:25<03:09, 2.23s/it]
Loading safetensors checkpoint shards: 12% Completed | 12/96 [00:27<03:02, 2.18s/it]
Loading safetensors checkpoint shards: 14% Completed | 13/96 [00:29<02:57, 2.14s/it]
Loading safetensors checkpoint shards: 15% Completed | 14/96 [00:31<02:52, 2.11s/it]
Loading safetensors checkpoint shards: 16% Completed | 15/96 [00:33<02:47, 2.07s/it]
Loading safetensors checkpoint shards: 17% Completed | 16/96 [00:35<02:45, 2.07s/it]
Loading safetensors checkpoint shards: 18% Completed | 17/96 [00:37<02:44, 2.08s/it]
Loading safetensors checkpoint shards: 19% Completed | 18/96 [00:39<02:41, 2.07s/it]
Loading safetensors checkpoint shards: 20% Completed | 19/96 [00:41<02:39, 2.07s/it]
Loading safetensors checkpoint shards: 21% Completed | 20/96 [00:43<02:37, 2.08s/it]
Loading safetensors checkpoint shards: 22% Completed | 21/96 [00:45<02:37, 2.09s/it]
Loading safetensors checkpoint shards: 23% Completed | 22/96 [00:48<02:43, 2.21s/it]
Loading safetensors checkpoint shards: 24% Completed | 23/96 [00:50<02:46, 2.28s/it]
Loading safetensors checkpoint shards: 25% Completed | 24/96 [00:53<02:46, 2.32s/it]
Loading safetensors checkpoint shards: 26% Completed | 25/96 [00:55<02:45, 2.34s/it]
Loading safetensors checkpoint shards: 27% Completed | 26/96 [00:57<02:44, 2.35s/it]
Loading safetensors checkpoint shards: 28% Completed | 27/96 [01:00<02:41, 2.34s/it]
Loading safetensors checkpoint shards: 29% Completed | 28/96 [01:02<02:39, 2.35s/it]
Loading safetensors checkpoint shards: 30% Completed | 29/96 [01:04<02:31, 2.26s/it]
Loading safetensors checkpoint shards: 31% Completed | 30/96 [01:06<02:29, 2.26s/it]
Loading safetensors checkpoint shards: 32% Completed | 31/96 [01:09<02:23, 2.21s/it]
Loading safetensors checkpoint shards: 33% Completed | 32/96 [01:11<02:18, 2.17s/it]
Loading safetensors checkpoint shards: 34% Completed | 33/96 [01:13<02:16, 2.17s/it]
Loading safetensors checkpoint shards: 35% Completed | 34/96 [01:15<02:13, 2.16s/it]
Loading safetensors checkpoint shards: 36% Completed | 35/96 [01:17<02:13, 2.19s/it]
Loading safetensors checkpoint shards: 38% Completed | 36/96 [01:19<02:13, 2.22s/it]
Loading safetensors checkpoint shards: 39% Completed | 37/96 [01:22<02:09, 2.20s/it]
Loading safetensors checkpoint shards: 40% Completed | 38/96 [01:24<02:12, 2.29s/it]
Loading safetensors checkpoint shards: 41% Completed | 39/96 [01:27<02:13, 2.35s/it]
Loading safetensors checkpoint shards: 42% Completed | 40/96 [01:29<02:13, 2.39s/it]
Loading safetensors checkpoint shards: 43% Completed | 41/96 [01:32<02:13, 2.42s/it]
Loading safetensors checkpoint shards: 44% Completed | 42/96 [01:34<02:09, 2.39s/it]
Loading safetensors checkpoint shards: 45% Completed | 43/96 [01:36<02:07, 2.40s/it]
Loading safetensors checkpoint shards: 46% Completed | 44/96 [01:39<02:04, 2.40s/it]
Loading safetensors checkpoint shards: 47% Completed | 45/96 [01:41<01:59, 2.35s/it]
Loading safetensors checkpoint shards: 48% Completed | 46/96 [01:43<01:52, 2.25s/it]
Loading safetensors checkpoint shards: 49% Completed | 47/96 [01:45<01:50, 2.25s/it]
Loading safetensors checkpoint shards: 50% Completed | 48/96 [01:47<01:48, 2.26s/it]
Loading safetensors checkpoint shards: 51% Completed | 49/96 [01:50<01:46, 2.27s/it]
Loading safetensors checkpoint shards: 52% Completed | 50/96 [01:52<01:44, 2.26s/it]
Loading safetensors checkpoint shards: 53% Completed | 51/96 [01:54<01:42, 2.27s/it]
Loading safetensors checkpoint shards: 54% Completed | 52/96 [01:57<01:40, 2.29s/it]
Loading safetensors checkpoint shards: 55% Completed | 53/96 [01:59<01:38, 2.29s/it]
Loading safetensors checkpoint shards: 56% Completed | 54/96 [02:01<01:36, 2.29s/it]
Loading safetensors checkpoint shards: 57% Completed | 55/96 [02:04<01:34, 2.31s/it]
Loading safetensors checkpoint shards: 58% Completed | 56/96 [02:06<01:32, 2.32s/it]
Loading safetensors checkpoint shards: 59% Completed | 57/96 [02:08<01:30, 2.31s/it]
Loading safetensors checkpoint shards: 60% Completed | 58/96 [02:11<01:28, 2.32s/it]
Loading safetensors checkpoint shards: 61% Completed | 59/96 [02:13<01:26, 2.33s/it]
Loading safetensors checkpoint shards: 62% Completed | 60/96 [02:15<01:22, 2.29s/it]
Loading safetensors checkpoint shards: 64% Completed | 61/96 [02:17<01:19, 2.28s/it]
Loading safetensors checkpoint shards: 65% Completed | 62/96 [02:20<01:18, 2.30s/it]
Loading safetensors checkpoint shards: 66% Completed | 63/96 [02:22<01:16, 2.30s/it]
Loading safetensors checkpoint shards: 67% Completed | 64/96 [02:24<01:14, 2.32s/it]
Loading safetensors checkpoint shards: 68% Completed | 65/96 [02:27<01:11, 2.32s/it]
Loading safetensors checkpoint shards: 69% Completed | 66/96 [02:29<01:09, 2.33s/it]
Loading safetensors checkpoint shards: 70% Completed | 67/96 [02:31<01:07, 2.33s/it]
Loading safetensors checkpoint shards: 71% Completed | 68/96 [02:34<01:05, 2.32s/it]
Loading safetensors checkpoint shards: 72% Completed | 69/96 [02:36<01:02, 2.33s/it]
Loading safetensors checkpoint shards: 73% Completed | 70/96 [02:38<01:00, 2.32s/it]
Loading safetensors checkpoint shards: 74% Completed | 71/96 [02:41<00:57, 2.29s/it]
Loading safetensors checkpoint shards: 75% Completed | 72/96 [02:43<00:54, 2.28s/it]
Loading safetensors checkpoint shards: 76% Completed | 73/96 [02:45<00:52, 2.27s/it]
Loading safetensors checkpoint shards: 77% Completed | 74/96 [02:47<00:50, 2.28s/it]
Loading safetensors checkpoint shards: 78% Completed | 75/96 [02:50<00:48, 2.30s/it]
Loading safetensors checkpoint shards: 79% Completed | 76/96 [02:52<00:46, 2.30s/it]
Loading safetensors checkpoint shards: 80% Completed | 77/96 [02:54<00:44, 2.35s/it]
Loading safetensors checkpoint shards: 81% Completed | 78/96 [02:57<00:43, 2.40s/it]
Loading safetensors checkpoint shards: 82% Completed | 79/96 [02:59<00:40, 2.40s/it]
Loading safetensors checkpoint shards: 83% Completed | 80/96 [03:02<00:38, 2.39s/it]
(Worker_TP2 pid=2190990) INFO 09-26 09:43:00 [default_loader.py:267] Loading weights took 184.29 seconds
(Worker_TP2 pid=2190990) INFO 09-26 09:43:01 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 184.943389 seconds
Loading safetensors checkpoint shards: 84% Completed | 81/96 [03:04<00:35, 2.37s/it]
Loading safetensors checkpoint shards: 85% Completed | 82/96 [03:07<00:33, 2.39s/it]
Loading safetensors checkpoint shards: 86% Completed | 83/96 [03:09<00:31, 2.40s/it]
Loading safetensors checkpoint shards: 88% Completed | 84/96 [03:11<00:28, 2.42s/it]
Loading safetensors checkpoint shards: 89% Completed | 85/96 [03:14<00:26, 2.41s/it]
Loading safetensors checkpoint shards: 90% Completed | 86/96 [03:16<00:24, 2.41s/it]
(Worker_TP3 pid=2190991) INFO 09-26 09:43:15 [default_loader.py:267] Loading weights took 198.12 seconds
Loading safetensors checkpoint shards: 91% Completed | 87/96 [03:18<00:20, 2.28s/it]
(Worker_TP3 pid=2190991) INFO 09-26 09:43:15 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 198.802290 seconds
(Worker_TP5 pid=2190993) INFO 09-26 09:43:16 [default_loader.py:267] Loading weights took 199.57 seconds
(Worker_TP5 pid=2190993) INFO 09-26 09:43:16 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 200.174205 seconds
Loading safetensors checkpoint shards: 92% Completed | 88/96 [03:20<00:17, 2.23s/it]
(Worker_TP1 pid=2190989) INFO 09-26 09:43:18 [default_loader.py:267] Loading weights took 201.89 seconds
(Worker_TP1 pid=2190989) INFO 09-26 09:43:18 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 202.575515 seconds
(Worker_TP7 pid=2190995) INFO 09-26 09:43:19 [default_loader.py:267] Loading weights took 202.24 seconds
Loading safetensors checkpoint shards: 93% Completed | 89/96 [03:22<00:15, 2.15s/it]
(Worker_TP7 pid=2190995) INFO 09-26 09:43:19 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 202.832375 seconds
Loading safetensors checkpoint shards: 94% Completed | 90/96 [03:24<00:12, 2.08s/it]
Loading safetensors checkpoint shards: 96% Completed | 92/96 [03:26<00:06, 1.58s/it]
Loading safetensors checkpoint shards: 97% Completed | 93/96 [03:28<00:05, 1.68s/it]
Loading safetensors checkpoint shards: 98% Completed | 94/96 [03:30<00:03, 1.80s/it]
Loading safetensors checkpoint shards: 99% Completed | 95/96 [03:32<00:01, 1.87s/it]
Loading safetensors checkpoint shards: 100% Completed | 96/96 [03:34<00:00, 1.90s/it]
Loading safetensors checkpoint shards: 100% Completed | 96/96 [03:34<00:00, 2.24s/it]
(Worker_TP0 pid=2190988)
(Worker_TP0 pid=2190988) INFO 09-26 09:43:31 [default_loader.py:267] Loading weights took 214.84 seconds
(Worker_TP0 pid=2190988) INFO 09-26 09:43:31 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 215.636401 seconds
(Worker_TP6 pid=2190994) INFO 09-26 09:43:33 [default_loader.py:267] Loading weights took 216.19 seconds
(Worker_TP6 pid=2190994) INFO 09-26 09:43:33 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 216.825923 seconds
(Worker_TP4 pid=2190992) INFO 09-26 09:43:35 [default_loader.py:267] Loading weights took 219.13 seconds
(Worker_TP4 pid=2190992) INFO 09-26 09:43:36 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 219.819439 seconds
(Worker_TP0 pid=2190988) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP6 pid=2190994) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP1 pid=2190989) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP2 pid=2190990) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP3 pid=2190991) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP5 pid=2190993) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP7 pid=2190995) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP4 pid=2190992) INFO 09-26 09:43:36 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP1 pid=2190989) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP5 pid=2190993) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP0 pid=2190988) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP7 pid=2190995) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP2 pid=2190990) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP4 pid=2190992) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP3 pid=2190991) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP6 pid=2190994) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP0 pid=2190988) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP1 pid=2190989) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP4 pid=2190992) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP5 pid=2190993) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP3 pid=2190991) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP6 pid=2190994) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP2 pid=2190990) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP7 pid=2190995) WARNING 09-26 09:44:25 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(EngineCore_DP0 pid=2190849) INFO 09-26 09:44:36 [shm_broadcast.py:458] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging, doing some time-consuming work (e.g. compilation), or sitting idle.
(Worker_TP0 pid=2190988) INFO 09-26 09:44:38 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2190988) INFO 09-26 09:44:38 [backends.py:559] Dynamo bytecode transform time: 13.19 s
(Worker_TP7 pid=2190995) INFO 09-26 09:44:39 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_7_0/backbone for vLLM's torch.compile
(Worker_TP7 pid=2190995) INFO 09-26 09:44:39 [backends.py:559] Dynamo bytecode transform time: 14.08 s
(Worker_TP2 pid=2190990) INFO 09-26 09:44:39 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_2_0/backbone for vLLM's torch.compile
(Worker_TP2 pid=2190990) INFO 09-26 09:44:39 [backends.py:559] Dynamo bytecode transform time: 14.28 s
(Worker_TP4 pid=2190992) INFO 09-26 09:44:40 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_4_0/backbone for vLLM's torch.compile
(Worker_TP4 pid=2190992) INFO 09-26 09:44:40 [backends.py:559] Dynamo bytecode transform time: 14.52 s
(Worker_TP1 pid=2190989) INFO 09-26 09:44:40 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_1_0/backbone for vLLM's torch.compile
(Worker_TP1 pid=2190989) INFO 09-26 09:44:40 [backends.py:559] Dynamo bytecode transform time: 14.81 s
(Worker_TP5 pid=2190993) INFO 09-26 09:44:40 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_5_0/backbone for vLLM's torch.compile
(Worker_TP5 pid=2190993) INFO 09-26 09:44:40 [backends.py:559] Dynamo bytecode transform time: 15.04 s
(Worker_TP6 pid=2190994) INFO 09-26 09:44:40 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_6_0/backbone for vLLM's torch.compile
(Worker_TP6 pid=2190994) INFO 09-26 09:44:40 [backends.py:559] Dynamo bytecode transform time: 15.11 s
(Worker_TP3 pid=2190991) INFO 09-26 09:44:41 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_3_0/backbone for vLLM's torch.compile
(Worker_TP3 pid=2190991) INFO 09-26 09:44:41 [backends.py:559] Dynamo bytecode transform time: 15.58 s
(Worker_TP0 pid=2190988) INFO 09-26 09:44:43 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP7 pid=2190995) INFO 09-26 09:44:44 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP2 pid=2190990) INFO 09-26 09:44:44 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP4 pid=2190992) INFO 09-26 09:44:45 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP1 pid=2190989) INFO 09-26 09:44:45 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP5 pid=2190993) INFO 09-26 09:44:45 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP6 pid=2190994) INFO 09-26 09:44:46 [backends.py:197] Cache the graph for dynamic shape for later use
(Worker_TP3 pid=2190991) INFO 09-26 09:44:47 [backends.py:197] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=2190849) INFO 09-26 09:45:36 [shm_broadcast.py:458] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging, doing some time-consuming work (e.g. compilation), or sitting idle.
(Worker_TP0 pid=2190988) INFO 09-26 09:46:06 [backends.py:218] Compiling a graph for dynamic shape takes 87.49 s
(Worker_TP7 pid=2190995) INFO 09-26 09:46:09 [backends.py:218] Compiling a graph for dynamic shape takes 89.06 s
(Worker_TP2 pid=2190990) INFO 09-26 09:46:14 [backends.py:218] Compiling a graph for dynamic shape takes 94.30 s
(Worker_TP1 pid=2190989) INFO 09-26 09:46:15 [backends.py:218] Compiling a graph for dynamic shape takes 94.37 s
(Worker_TP5 pid=2190993) INFO 09-26 09:46:16 [backends.py:218] Compiling a graph for dynamic shape takes 94.83 s
(Worker_TP4 pid=2190992) INFO 09-26 09:46:16 [backends.py:218] Compiling a graph for dynamic shape takes 96.14 s
(Worker_TP6 pid=2190994) INFO 09-26 09:46:26 [backends.py:218] Compiling a graph for dynamic shape takes 104.72 s
(Worker_TP3 pid=2190991) INFO 09-26 09:46:29 [backends.py:218] Compiling a graph for dynamic shape takes 107.65 s
(Worker_TP6 pid=2190994) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP5 pid=2190993) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP0 pid=2190988) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP7 pid=2190995) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP3 pid=2190991) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP2 pid=2190990) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP1 pid=2190989) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP4 pid=2190992) WARNING 09-26 09:46:31 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP6 pid=2190994) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 119.83 s in total
(Worker_TP5 pid=2190993) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 109.86 s in total
(Worker_TP0 pid=2190988) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 100.69 s in total
(Worker_TP7 pid=2190995) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 103.14 s in total
(Worker_TP4 pid=2190992) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 110.65 s in total
(Worker_TP2 pid=2190990) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 108.58 s in total
(Worker_TP1 pid=2190989) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 109.19 s in total
(Worker_TP3 pid=2190991) INFO 09-26 09:46:32 [monitor.py:34] torch.compile takes 123.23 s in total
(Worker_TP7 pid=2190995) INFO 09-26 09:46:33 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP0 pid=2190988) INFO 09-26 09:46:33 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP6 pid=2190994) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP1 pid=2190989) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP2 pid=2190990) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP5 pid=2190993) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP4 pid=2190992) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(Worker_TP3 pid=2190991) INFO 09-26 09:46:34 [gpu_worker.py:298] Available KV cache memory: 34.78 GiB
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1087] GPU KV cache size: 775,904 tokens
(EngineCore_DP0 pid=2190849) INFO 09-26 09:46:35 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(Worker_TP0 pid=2190988) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP7 pid=2190995) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP4 pid=2190992) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP2 pid=2190990) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP6 pid=2190994) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP1 pid=2190989) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP3 pid=2190991) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP5 pid=2190993) INFO 09-26 09:46:35 [utils.py:351] _KV_CACHE_LAYOUT_OVERRIDE variable detected. Setting KV cache layout to HND.
(Worker_TP0 pid=2190988) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP5 pid=2190993) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2190989) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2190990) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP7 pid=2190995) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP4 pid=2190992) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2190991) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP6 pid=2190994) 2025-09-26 09:46:35,229 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP7 pid=2190995) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2190988) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP4 pid=2190992) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2190991) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2190989) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=2190990) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP5 pid=2190993) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP7 pid=2190995) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP0 pid=2190988) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP6 pid=2190994) 2025-09-26 09:46:35,590 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP4 pid=2190992) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP3 pid=2190991) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP1 pid=2190989) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP2 pid=2190990) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP6 pid=2190994) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
(Worker_TP5 pid=2190993) INFO 09-26 09:46:35 [kernel_warmup.py:52] Warming up FlashInfer attention.
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:19<00:00, 3.45it/s]
Capturing CUDA graphs (decode, FULL): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:19<00:00, 3.49it/s]
(Worker_TP0 pid=2190988) INFO 09-26 09:47:15 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP7 pid=2190995) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP5 pid=2190993) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP1 pid=2190989) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP6 pid=2190994) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP2 pid=2190990) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP4 pid=2190992) INFO 09-26 09:47:16 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP3 pid=2190991) INFO 09-26 09:47:17 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP7 pid=2190995) INFO 09-26 09:47:17 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP0 pid=2190988) INFO 09-26 09:47:17 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP5 pid=2190993) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP6 pid=2190994) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP2 pid=2190990) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP3 pid=2190991) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP4 pid=2190992) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(Worker_TP1 pid=2190989) INFO 09-26 09:47:18 [gpu_model_runner.py:3379] Graph capturing finished in 41 secs, took 1.18 GiB
(EngineCore_DP0 pid=2190849) INFO 09-26 09:47:18 [core.py:210] init engine (profile, create kv cache, warmup model) took 222.20 seconds
INFO 09-26 09:47:24 [llm.py:302] Supported_tasks: ['generate']

========================================
Inputs[0]: input_['prompt']='<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Read all the text in the image.<|im_end|>\n<|im_start|>assistant\n'

Adding requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00, 5.04s/it]
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](Worker_TP7 pid=2190995) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP0 pid=2190988) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP5 pid=2190993) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP1 pid=2190989) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP3 pid=2190991) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP4 pid=2190992) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP2 pid=2190990) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
(Worker_TP6 pid=2190994) WARNING 09-26 09:47:29 [flashinfer.py:280] Using TRTLLM attention (auto-detected).
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:15<00:00, 15.05s/it, est. speed input: 16.34 toks/s, output: 68.04 toks/s]

========================================
Generated text: '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
(Worker_TP0 pid=2190988) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP0 pid=2190988) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP1 pid=2190989) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP1 pid=2190989) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP2 pid=2190990) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP2 pid=2190990) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP3 pid=2190991) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP3 pid=2190991) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP4 pid=2190992) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP5 pid=2190993) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP4 pid=2190992) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP5 pid=2190993) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP6 pid=2190994) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP6 pid=2190994) INFO 09-26 09:47:44 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP7 pid=2190995) INFO 09-26 09:47:44 [multiproc_executor.py:558] Parent process exited, terminating worker

Offline Inference with remove flashinfer logs:
(kta) kta@kta-02:~$ python test_qwen3_vl.py
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:48:54 [init.py:216] Automatically detected platform cuda.
video_kwargs: {'do_sample_frames': False}
INFO 09-26 09:49:34 [utils.py:233] non-default args: {'trust_remote_code': True, 'seed': 0, 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.7, 'disable_log_stats': True, 'model': '/data/qwen3-vl'}
The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
INFO 09-26 09:49:34 [model.py:540] Resolved architecture: Qwen3VLMoeForConditionalGeneration
torch_dtype is deprecated! Use dtype instead!
INFO 09-26 09:49:34 [model.py:1502] Using max model len 262144
INFO 09-26 09:49:34 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=16384.
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:49:38 [init.py:216] Automatically detected platform cuda.
(EngineCore_DP0 pid=2204345) INFO 09-26 09:49:39 [core.py:644] Waiting for init message from front-end.
(EngineCore_DP0 pid=2204345) INFO 09-26 09:49:39 [core.py:77] Initializing a V1 LLM engine (v0.11.0rc2.dev131+g081b5594a) with config: model='/data/qwen3-vl', speculative_config=None, tokenizer='/data/qwen3-vl', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/data/qwen3-vl, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
(EngineCore_DP0 pid=2204345) WARNING 09-26 09:49:39 [multiproc_executor.py:720] Reducing Torch parallelism from 144 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore_DP0 pid=2204345) INFO 09-26 09:49:39 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3, 4, 5, 6, 7], buffer_handle=(8, 16777216, 10, 'psm_5a4ad7d5'), local_subscribe_addr='ipc:///tmp/b5a7f524-69c9-4659-ae70-729af7485867', remote_subscribe_addr=None, remote_addr_ipv6=False)
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
/home/kta/.venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
INFO 09-26 09:49:43 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:43 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:43 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:43 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:44 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:44 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:44 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:44 [init.py:216] Automatically detected platform cuda.
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_10a12958'), local_subscribe_addr='ipc:///tmp/36da3e68-3887-4a22-9bef-8955a06955a5', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_7cae4f2a'), local_subscribe_addr='ipc:///tmp/3f778b9d-be03-4796-b722-06d4012a1487', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_4573cbc3'), local_subscribe_addr='ipc:///tmp/ef38902a-cc58-474b-b6ba-827d6fbdba4c', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_5c8573b7'), local_subscribe_addr='ipc:///tmp/658b4d96-e3f1-47a7-87d1-99409dd10901', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_e2334a58'), local_subscribe_addr='ipc:///tmp/d96c77d4-0f4a-4800-95d0-ad352a33dc4a', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_fa7dbc25'), local_subscribe_addr='ipc:///tmp/d964f556-0bd5-4cd8-8a7e-aa42b8679279', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d8d6b8a2'), local_subscribe_addr='ipc:///tmp/0dccebee-1ae0-4666-8891-539490a49cc3', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 09-26 09:49:56 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_863811a8'), local_subscribe_addr='ipc:///tmp/37c98390-2f15-4bce-a7d4-13dd7575e0df', remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:49:58 [init.py:1382] Found nccl from library libnccl.so.2
INFO 09-26 09:49:58 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [custom_all_reduce.py:35] Skipping P2P check and trusting the driver's P2P report.
INFO 09-26 09:50:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_33a08fe7'), local_subscribe_addr='ipc:///tmp/0e9b1926-a5cd-4e9d-a085-3fe73664ec2e', remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
INFO 09-26 09:50:03 [parallel_state.py:1201] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
WARNING 09-26 09:50:04 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(Worker_TP6 pid=2204490) INFO 09-26 09:50:06 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP3 pid=2204487) INFO 09-26 09:50:06 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP3 pid=2204487) INFO 09-26 09:50:06 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP7 pid=2204491) INFO 09-26 09:50:06 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP6 pid=2204490) INFO 09-26 09:50:06 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP3 pid=2204487) WARNING 09-26 09:50:06 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP3 pid=2204487) INFO 09-26 09:50:06 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP6 pid=2204490) WARNING 09-26 09:50:06 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP6 pid=2204490) INFO 09-26 09:50:06 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP7 pid=2204491) INFO 09-26 09:50:06 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP7 pid=2204491) WARNING 09-26 09:50:06 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP7 pid=2204491) INFO 09-26 09:50:06 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP0 pid=2204484) INFO 09-26 09:50:07 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP4 pid=2204488) INFO 09-26 09:50:07 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP5 pid=2204489) INFO 09-26 09:50:07 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP2 pid=2204486) INFO 09-26 09:50:07 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP1 pid=2204485) INFO 09-26 09:50:07 [gpu_model_runner.py:2536] Starting to load model /data/qwen3-vl...
(Worker_TP0 pid=2204484) INFO 09-26 09:50:07 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP4 pid=2204488) INFO 09-26 09:50:07 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP0 pid=2204484) WARNING 09-26 09:50:07 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP0 pid=2204484) INFO 09-26 09:50:07 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP4 pid=2204488) WARNING 09-26 09:50:07 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP4 pid=2204488) INFO 09-26 09:50:07 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP5 pid=2204489) INFO 09-26 09:50:07 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP2 pid=2204486) INFO 09-26 09:50:07 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP1 pid=2204485) INFO 09-26 09:50:07 [gpu_model_runner.py:2568] Loading model from scratch...
(Worker_TP5 pid=2204489) WARNING 09-26 09:50:07 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP5 pid=2204489) INFO 09-26 09:50:07 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP2 pid=2204486) WARNING 09-26 09:50:07 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP2 pid=2204486) INFO 09-26 09:50:07 [cuda.py:347] Using Flash Attention backend on V1 engine.
(Worker_TP1 pid=2204485) WARNING 09-26 09:50:07 [cuda.py:333] FlashInfer failed to import for V1 engine on Blackwell (SM 10.0) GPUs; it is recommended to install FlashInfer for better performance.
(Worker_TP1 pid=2204485) INFO 09-26 09:50:07 [cuda.py:347] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 0% Completed | 0/96 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 2% Completed | 2/96 [00:01<01:22, 1.14it/s]
Loading safetensors checkpoint shards: 3% Completed | 3/96 [00:03<01:51, 1.20s/it]
Loading safetensors checkpoint shards: 4% Completed | 4/96 [00:05<02:05, 1.37s/it]
Loading safetensors checkpoint shards: 5% Completed | 5/96 [00:06<02:15, 1.49s/it]
Loading safetensors checkpoint shards: 6% Completed | 6/96 [00:08<02:29, 1.66s/it]
Loading safetensors checkpoint shards: 7% Completed | 7/96 [00:11<02:46, 1.87s/it]
Loading safetensors checkpoint shards: 8% Completed | 8/96 [00:13<02:54, 1.99s/it]
Loading safetensors checkpoint shards: 9% Completed | 9/96 [00:15<02:58, 2.05s/it]
Loading safetensors checkpoint shards: 10% Completed | 10/96 [00:17<03:00, 2.10s/it]
Loading safetensors checkpoint shards: 11% Completed | 11/96 [00:20<03:14, 2.29s/it]
Loading safetensors checkpoint shards: 12% Completed | 12/96 [00:22<03:10, 2.26s/it]
Loading safetensors checkpoint shards: 14% Completed | 13/96 [00:24<03:07, 2.25s/it]
Loading safetensors checkpoint shards: 15% Completed | 14/96 [00:27<03:04, 2.25s/it]
Loading safetensors checkpoint shards: 16% Completed | 15/96 [00:29<03:03, 2.27s/it]
Loading safetensors checkpoint shards: 17% Completed | 16/96 [00:31<03:00, 2.25s/it]
Loading safetensors checkpoint shards: 18% Completed | 17/96 [00:33<02:57, 2.25s/it]
Loading safetensors checkpoint shards: 19% Completed | 18/96 [00:36<02:54, 2.24s/it]
Loading safetensors checkpoint shards: 20% Completed | 19/96 [00:38<02:52, 2.24s/it]
Loading safetensors checkpoint shards: 21% Completed | 20/96 [00:40<02:50, 2.24s/it]
Loading safetensors checkpoint shards: 22% Completed | 21/96 [00:42<02:47, 2.24s/it]
Loading safetensors checkpoint shards: 23% Completed | 22/96 [00:44<02:41, 2.18s/it]
Loading safetensors checkpoint shards: 24% Completed | 23/96 [00:46<02:37, 2.16s/it]
Loading safetensors checkpoint shards: 25% Completed | 24/96 [00:49<02:34, 2.14s/it]
Loading safetensors checkpoint shards: 26% Completed | 25/96 [00:51<02:31, 2.13s/it]
Loading safetensors checkpoint shards: 27% Completed | 26/96 [00:53<02:29, 2.14s/it]
Loading safetensors checkpoint shards: 28% Completed | 27/96 [00:55<02:30, 2.18s/it]
Loading safetensors checkpoint shards: 29% Completed | 28/96 [00:57<02:27, 2.17s/it]
Loading safetensors checkpoint shards: 30% Completed | 29/96 [01:00<02:30, 2.24s/it]
Loading safetensors checkpoint shards: 31% Completed | 30/96 [01:02<02:27, 2.24s/it]
Loading safetensors checkpoint shards: 32% Completed | 31/96 [01:04<02:27, 2.28s/it]
Loading safetensors checkpoint shards: 33% Completed | 32/96 [01:07<02:27, 2.31s/it]
Loading safetensors checkpoint shards: 34% Completed | 33/96 [01:09<02:26, 2.33s/it]
Loading safetensors checkpoint shards: 35% Completed | 34/96 [01:11<02:26, 2.36s/it]
Loading safetensors checkpoint shards: 36% Completed | 35/96 [01:14<02:22, 2.34s/it]
Loading safetensors checkpoint shards: 38% Completed | 36/96 [01:16<02:19, 2.32s/it]
Loading safetensors checkpoint shards: 39% Completed | 37/96 [01:18<02:17, 2.34s/it]
Loading safetensors checkpoint shards: 40% Completed | 38/96 [01:20<02:11, 2.26s/it]
Loading safetensors checkpoint shards: 41% Completed | 39/96 [01:23<02:05, 2.21s/it]
Loading safetensors checkpoint shards: 42% Completed | 40/96 [01:25<02:01, 2.16s/it]
Loading safetensors checkpoint shards: 43% Completed | 41/96 [01:27<02:00, 2.20s/it]
Loading safetensors checkpoint shards: 44% Completed | 42/96 [01:29<02:01, 2.25s/it]
Loading safetensors checkpoint shards: 45% Completed | 43/96 [01:31<01:57, 2.22s/it]
Loading safetensors checkpoint shards: 46% Completed | 44/96 [01:33<01:52, 2.17s/it]
Loading safetensors checkpoint shards: 47% Completed | 45/96 [01:35<01:47, 2.12s/it]
Loading safetensors checkpoint shards: 48% Completed | 46/96 [01:37<01:44, 2.09s/it]
Loading safetensors checkpoint shards: 49% Completed | 47/96 [01:39<01:38, 2.02s/it]
Loading safetensors checkpoint shards: 50% Completed | 48/96 [01:41<01:33, 1.95s/it]
Loading safetensors checkpoint shards: 51% Completed | 49/96 [01:43<01:29, 1.91s/it]
Loading safetensors checkpoint shards: 52% Completed | 50/96 [01:45<01:28, 1.93s/it]
Loading safetensors checkpoint shards: 53% Completed | 51/96 [01:47<01:30, 2.01s/it]
Loading safetensors checkpoint shards: 54% Completed | 52/96 [01:49<01:32, 2.10s/it]
Loading safetensors checkpoint shards: 55% Completed | 53/96 [01:51<01:29, 2.08s/it]
Loading safetensors checkpoint shards: 56% Completed | 54/96 [01:54<01:27, 2.07s/it]
Loading safetensors checkpoint shards: 57% Completed | 55/96 [01:55<01:22, 2.02s/it]
Loading safetensors checkpoint shards: 58% Completed | 56/96 [01:57<01:19, 1.99s/it]
Loading safetensors checkpoint shards: 59% Completed | 57/96 [01:59<01:15, 1.92s/it]
Loading safetensors checkpoint shards: 60% Completed | 58/96 [02:01<01:16, 2.02s/it]
Loading safetensors checkpoint shards: 61% Completed | 59/96 [02:03<01:13, 1.99s/it]
Loading safetensors checkpoint shards: 62% Completed | 60/96 [02:05<01:08, 1.90s/it]
Loading safetensors checkpoint shards: 64% Completed | 61/96 [02:07<01:04, 1.85s/it]
Loading safetensors checkpoint shards: 65% Completed | 62/96 [02:09<01:07, 1.99s/it]
Loading safetensors checkpoint shards: 66% Completed | 63/96 [02:11<01:09, 2.10s/it]
Loading safetensors checkpoint shards: 67% Completed | 64/96 [02:14<01:08, 2.13s/it]
Loading safetensors checkpoint shards: 68% Completed | 65/96 [02:15<01:02, 2.03s/it]
Loading safetensors checkpoint shards: 69% Completed | 66/96 [02:17<00:58, 1.97s/it]
Loading safetensors checkpoint shards: 70% Completed | 67/96 [02:19<00:55, 1.92s/it]
Loading safetensors checkpoint shards: 71% Completed | 68/96 [02:21<00:52, 1.88s/it]
Loading safetensors checkpoint shards: 72% Completed | 69/96 [02:23<00:50, 1.86s/it]
Loading safetensors checkpoint shards: 73% Completed | 70/96 [02:24<00:48, 1.85s/it]
Loading safetensors checkpoint shards: 74% Completed | 71/96 [02:27<00:49, 1.99s/it]
Loading safetensors checkpoint shards: 75% Completed | 72/96 [02:29<00:49, 2.07s/it]
Loading safetensors checkpoint shards: 76% Completed | 73/96 [02:31<00:48, 2.12s/it]
Loading safetensors checkpoint shards: 77% Completed | 74/96 [02:34<00:47, 2.17s/it]
Loading safetensors checkpoint shards: 78% Completed | 75/96 [02:36<00:46, 2.19s/it]
Loading safetensors checkpoint shards: 79% Completed | 76/96 [02:38<00:44, 2.21s/it]
Loading safetensors checkpoint shards: 80% Completed | 77/96 [02:40<00:42, 2.25s/it]
Loading safetensors checkpoint shards: 81% Completed | 78/96 [02:43<00:41, 2.30s/it]
Loading safetensors checkpoint shards: 82% Completed | 79/96 [02:45<00:39, 2.31s/it]
Loading safetensors checkpoint shards: 83% Completed | 80/96 [02:47<00:36, 2.31s/it]
Loading safetensors checkpoint shards: 84% Completed | 81/96 [02:50<00:34, 2.31s/it]
(Worker_TP3 pid=2204487) INFO 09-26 09:52:58 [default_loader.py:267] Loading weights took 171.24 seconds
(Worker_TP3 pid=2204487) INFO 09-26 09:52:58 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 171.494605 seconds
Loading safetensors checkpoint shards: 85% Completed | 82/96 [02:52<00:32, 2.34s/it]
Loading safetensors checkpoint shards: 86% Completed | 83/96 [02:54<00:30, 2.34s/it]
(Worker_TP6 pid=2204490) INFO 09-26 09:53:04 [default_loader.py:267] Loading weights took 177.25 seconds
(Worker_TP6 pid=2204490) INFO 09-26 09:53:04 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 177.501356 seconds
Loading safetensors checkpoint shards: 88% Completed | 84/96 [02:57<00:28, 2.35s/it]
Loading safetensors checkpoint shards: 89% Completed | 85/96 [02:59<00:25, 2.36s/it]
Loading safetensors checkpoint shards: 90% Completed | 86/96 [03:02<00:23, 2.36s/it]
Loading safetensors checkpoint shards: 91% Completed | 87/96 [03:04<00:20, 2.23s/it]
Loading safetensors checkpoint shards: 92% Completed | 88/96 [03:06<00:17, 2.19s/it]
Loading safetensors checkpoint shards: 93% Completed | 89/96 [03:08<00:14, 2.11s/it]
Loading safetensors checkpoint shards: 94% Completed | 90/96 [03:09<00:12, 2.04s/it]
Loading safetensors checkpoint shards: 96% Completed | 92/96 [03:11<00:06, 1.55s/it]
Loading safetensors checkpoint shards: 97% Completed | 93/96 [03:13<00:04, 1.65s/it]
Loading safetensors checkpoint shards: 98% Completed | 94/96 [03:15<00:03, 1.76s/it]
(Worker_TP7 pid=2204491) INFO 09-26 09:53:23 [default_loader.py:267] Loading weights took 196.51 seconds
(Worker_TP7 pid=2204491) INFO 09-26 09:53:24 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 196.750958 seconds
Loading safetensors checkpoint shards: 99% Completed | 95/96 [03:17<00:01, 1.84s/it]
Loading safetensors checkpoint shards: 100% Completed | 96/96 [03:19<00:00, 1.87s/it]
Loading safetensors checkpoint shards: 100% Completed | 96/96 [03:19<00:00, 2.08s/it]
(Worker_TP0 pid=2204484)
(Worker_TP0 pid=2204484) INFO 09-26 09:53:27 [default_loader.py:267] Loading weights took 199.96 seconds
(Worker_TP0 pid=2204484) INFO 09-26 09:53:28 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 200.256766 seconds
(Worker_TP2 pid=2204486) INFO 09-26 09:53:29 [default_loader.py:267] Loading weights took 201.85 seconds
(Worker_TP2 pid=2204486) INFO 09-26 09:53:30 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 202.154052 seconds
(Worker_TP5 pid=2204489) INFO 09-26 09:53:34 [default_loader.py:267] Loading weights took 206.74 seconds
(Worker_TP5 pid=2204489) INFO 09-26 09:53:34 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 207.054273 seconds
(Worker_TP4 pid=2204488) INFO 09-26 09:53:51 [default_loader.py:267] Loading weights took 224.24 seconds
(Worker_TP4 pid=2204488) INFO 09-26 09:53:52 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 224.538727 seconds
(Worker_TP1 pid=2204485) INFO 09-26 09:53:58 [default_loader.py:267] Loading weights took 231.04 seconds
(Worker_TP1 pid=2204485) INFO 09-26 09:53:59 [gpu_model_runner.py:2587] Model loading took 55.6785 GiB and 231.345660 seconds
(Worker_TP0 pid=2204484) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP2 pid=2204486) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP3 pid=2204487) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP5 pid=2204489) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP7 pid=2204491) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP1 pid=2204485) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP6 pid=2204490) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP4 pid=2204488) INFO 09-26 09:53:59 [gpu_model_runner.py:3260] Encoder cache will be initialized with a budget of 262144 tokens, and profiled with 1 video items of the maximum feature size.
(Worker_TP6 pid=2204490) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP0 pid=2204484) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP3 pid=2204487) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP5 pid=2204489) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP7 pid=2204491) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP2 pid=2204486) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP1 pid=2204485) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP4 pid=2204488) Token indices sequence length is longer than the specified maximum sequence length for this model (262278 > 262144). Running this sequence through the model will result in indexing errors
(Worker_TP0 pid=2204484) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP6 pid=2204490) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP2 pid=2204486) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP5 pid=2204489) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP3 pid=2204487) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP7 pid=2204491) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP1 pid=2204485) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP4 pid=2204488) WARNING 09-26 09:54:43 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
(Worker_TP7 pid=2204491) INFO 09-26 09:54:57 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_7_0/backbone for vLLM's torch.compile
(Worker_TP7 pid=2204491) INFO 09-26 09:54:57 [backends.py:559] Dynamo bytecode transform time: 13.33 s
(Worker_TP0 pid=2204484) INFO 09-26 09:54:57 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2204484) INFO 09-26 09:54:57 [backends.py:559] Dynamo bytecode transform time: 13.47 s
(Worker_TP5 pid=2204489) INFO 09-26 09:54:58 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_5_0/backbone for vLLM's torch.compile
(Worker_TP5 pid=2204489) INFO 09-26 09:54:58 [backends.py:559] Dynamo bytecode transform time: 14.41 s
(Worker_TP6 pid=2204490) INFO 09-26 09:54:58 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_6_0/backbone for vLLM's torch.compile
(Worker_TP6 pid=2204490) INFO 09-26 09:54:58 [backends.py:559] Dynamo bytecode transform time: 14.50 s
(Worker_TP3 pid=2204487) INFO 09-26 09:54:58 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_3_0/backbone for vLLM's torch.compile
(Worker_TP3 pid=2204487) INFO 09-26 09:54:58 [backends.py:559] Dynamo bytecode transform time: 14.62 s
(Worker_TP1 pid=2204485) INFO 09-26 09:54:58 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_1_0/backbone for vLLM's torch.compile
(Worker_TP1 pid=2204485) INFO 09-26 09:54:58 [backends.py:559] Dynamo bytecode transform time: 14.67 s
(Worker_TP2 pid=2204486) INFO 09-26 09:54:58 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_2_0/backbone for vLLM's torch.compile
(Worker_TP2 pid=2204486) INFO 09-26 09:54:58 [backends.py:559] Dynamo bytecode transform time: 14.80 s
(EngineCore_DP0 pid=2204345) INFO 09-26 09:54:59 [shm_broadcast.py:458] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging, doing some time-consuming work (e.g. compilation), or sitting idle.
(Worker_TP4 pid=2204488) INFO 09-26 09:54:59 [backends.py:548] Using cache directory: /home/kta/.cache/vllm/torch_compile_cache/9f40d11338/rank_4_0/backbone for vLLM's torch.compile
(Worker_TP4 pid=2204488) INFO 09-26 09:54:59 [backends.py:559] Dynamo bytecode transform time: 15.81 s
(Worker_TP7 pid=2204491) INFO 09-26 09:55:02 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.543 s
(Worker_TP0 pid=2204484) INFO 09-26 09:55:02 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.531 s
(Worker_TP5 pid=2204489) INFO 09-26 09:55:03 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.871 s
(Worker_TP6 pid=2204490) INFO 09-26 09:55:03 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.889 s
(Worker_TP3 pid=2204487) INFO 09-26 09:55:04 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.928 s
(Worker_TP2 pid=2204486) INFO 09-26 09:55:04 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 4.910 s
(Worker_TP1 pid=2204485) INFO 09-26 09:55:04 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 5.523 s
(Worker_TP4 pid=2204488) INFO 09-26 09:55:05 [backends.py:164] Directly load the compiled graph(s) for dynamic shape from the cache, took 5.384 s
(Worker_TP7 pid=2204491) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP0 pid=2204484) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP3 pid=2204487) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP4 pid=2204488) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP5 pid=2204489) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP2 pid=2204486) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP6 pid=2204490) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP1 pid=2204485) WARNING 09-26 09:55:07 [fused_moe.py:798] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/home/kta/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_B200.json']
(Worker_TP0 pid=2204484) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 13.47 s in total
(Worker_TP7 pid=2204491) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 13.33 s in total
(Worker_TP5 pid=2204489) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 14.41 s in total
(Worker_TP2 pid=2204486) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 14.80 s in total
(Worker_TP3 pid=2204487) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 14.62 s in total
(Worker_TP6 pid=2204490) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 14.50 s in total
(Worker_TP4 pid=2204488) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 15.81 s in total
(Worker_TP1 pid=2204485) INFO 09-26 09:55:07 [monitor.py:34] torch.compile takes 14.67 s in total
(Worker_TP7 pid=2204491) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP0 pid=2204484) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP5 pid=2204489) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP4 pid=2204488) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP6 pid=2204490) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP3 pid=2204487) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP2 pid=2204486) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(Worker_TP1 pid=2204485) INFO 09-26 09:55:08 [gpu_worker.py:298] Available KV cache memory: 34.77 GiB
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1087] GPU KV cache size: 775,776 tokens
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:08 [kv_cache_utils.py:1091] Maximum concurrency for 262,144 tokens per request: 2.96x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:17<00:00, 3.78it/s]
Capturing CUDA graphs (decode, FULL): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:17<00:00, 3.85it/s]
(Worker_TP0 pid=2204484) INFO 09-26 09:55:44 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP7 pid=2204491) INFO 09-26 09:55:44 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP6 pid=2204490) INFO 09-26 09:55:44 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP3 pid=2204487) INFO 09-26 09:55:45 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP2 pid=2204486) INFO 09-26 09:55:45 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP5 pid=2204489) INFO 09-26 09:55:45 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP4 pid=2204488) INFO 09-26 09:55:45 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP1 pid=2204485) INFO 09-26 09:55:45 [custom_all_reduce.py:203] Registering 6768 cuda graph addresses
(Worker_TP0 pid=2204484) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP7 pid=2204491) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP3 pid=2204487) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP6 pid=2204490) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP5 pid=2204489) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP2 pid=2204486) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP4 pid=2204488) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(Worker_TP1 pid=2204485) INFO 09-26 09:55:46 [gpu_model_runner.py:3379] Graph capturing finished in 37 secs, took -0.66 GiB
(EngineCore_DP0 pid=2204345) INFO 09-26 09:55:46 [core.py:210] init engine (profile, create kv cache, warmup model) took 107.40 seconds
INFO 09-26 09:55:52 [llm.py:302] Supported_tasks: ['generate']

========================================
Inputs[0]: input_['prompt']='<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Read all the text in the image.<|im_end|>\n<|im_start|>assistant\n'

Adding requests: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00, 5.67s/it]
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.73s/it, est. speed input: 14.70 toks/s, output: 61.20 toks/s]

========================================
Generated text: 'BATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nBATTLE\nB'
(Worker_TP0 pid=2204484) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP0 pid=2204484) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP1 pid=2204485) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP1 pid=2204485) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP2 pid=2204486) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP2 pid=2204486) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP3 pid=2204487) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP3 pid=2204487) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP4 pid=2204488) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP4 pid=2204488) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP5 pid=2204489) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP5 pid=2204489) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.
(Worker_TP6 pid=2204490) INFO 09-26 09:56:15 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP6 pid=2204490) INFO 09-26 09:56:15 [multiproc_executor.py:599] WorkerProc shutting down.

[ywang96]

@KTA-LAB just to confirm - does this issue only happen on B200? I don't think we have tested this model on blackwell so that could be where the culprit is.