[Bug]: some error when training in ray

[Jacoblincc]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.6.0+cu124
Is debug build               : False
CUDA used to build PyTorch   : 12.4
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform              : Linux-5.15.0-116-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 550.54.15
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8462Y+
CPU family:                           6
Model:                                143
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             8
Frequency boost:                      enabled
CPU max MHz:                          2801.0000
CPU min MHz:                          800.0000
BogoMIPS:                             5600.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            3 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             128 MiB (64 instances)
L3 cache:                             120 MiB (2 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-31,64-95
NUMA node1 CPU(s):                    32-63,96-127
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.1.3
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==27.0.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchdata==0.11.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.52.4
[pip3] triton==3.2.0
[conda] numpy                     2.1.3                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.2                    pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     27.0.0                   pypi_0    pypi
[conda] torch                     2.6.0                    pypi_0    pypi
[conda] torchaudio                2.6.0                    pypi_0    pypi
[conda] torchdata                 0.11.0                   pypi_0    pypi
[conda] torchvision               0.21.0                   pypi_0    pypi
[conda] transformers              4.52.4                   pypi_0    pypi
[conda] triton                    3.2.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.8.3
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     32-63,96-127    1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     32-63,96-127    1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     32-63,96-127    1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     32-63,96-127    1               N/A
NIC0    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC2    SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC3    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS     SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS     SYS     SYS     SYS     SYS
NIC5    SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS
NIC6    SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS
NIC7    SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS
NIC8    SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS
NIC9    SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9

==============================
     Environment Variables
==============================
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

when I use the verl to for training, sometimes a error occured:

Fatal Python error: none_dealloc: deallocating None
�[36m(WorkerDict pid=494481)�[0m Python runtime state: initialized
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Thread 0x00007f36860ff640 (most recent call first):
�[36m(WorkerDict pid=494481)�[0m   <no Python frame>
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Thread 0x00007f3687bfd640 (most recent call first):
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/usage/usage_lib.py", line 229 in _report_continous_usage
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/usage/usage_lib.py", line 164 in _report_usage_worker
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 946 in run
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 966 in _bootstrap
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Thread 0x00007f37cd7fe640 (most recent call first):
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 53 in _recv_msg
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 161 in _read_thread
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 946 in run
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 966 in _bootstrap
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Thread 0x00007f3800ddd640 (most recent call first):
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 324 in wait
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 600 in wait
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/threading.py", line 966 in _bootstrap
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Current thread 0x00007f6b14dda740 (most recent call first):
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/device_allocator/cumem.py", line 81 in unmap_and_release
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/device_allocator/cumem.py", line 206 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 90 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/utils.py", line 2456 in run_method
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56 in collective_rpc
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 206 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 268 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 220 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 245 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1254 in sleep
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/workers/sharding_manager/fsdp_vllm.py", line 216 in __exit__
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/utils/debug/performance.py", line 88 in log
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/utils/debug/performance.py", line 78 in f
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/workers/fsdp_workers.py", line 692 in generate_sequences
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/single_controller/base/decorator.py", line 540 in inner
�[36m(WorkerDict pid=494481)�[0m   File "/data01/linxiang/verl/verl/single_controller/ray/base.py", line 645 in func
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py", line 463 in _resume_span
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/function_manager.py", line 689 in actor_method_executor
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/worker.py", line 953 in main_loop
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/workers/default_worker.py", line 330 in <module>
�[36m(WorkerDict pid=494481)�[0m 
�[36m(WorkerDict pid=494481)�[0m Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, regex._regex, markupsafe._speedups, PIL._imaging, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, PIL._imagingft, pyarrow._json, zmq.backend.cython._zmq, msgspec._core, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, sentencepiece._sentencepiece, vllm.cumem_allocator, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.typing.builtins.itertools, numba.cpython.builtins.math, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box (total: 157)
�[33m(raylet)�[0m A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff7710216a6912f94f21d0dafe01000000 Worker ID: a3633a269a5f0c238cab926642fc00db4d898cc87b76fa7c125ddb7b Node ID: 995f2037a22f7f703ce135becd5d1eddb50454abcbe62dd8b9c6bd9b Worker IP address: 10.0.30.111 Worker port: 44415 Worker PID: 494481 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
Error executing job with overrides: ['data.train_files=/data01/linxiang/datasets/Merge_COT/grpo_merge_all_0618.jsonl', 'data.val_files=/data01/linxiang/datasets/Merge_COT/dapo_merge_all_0618_val.jsonl', 'data.prompt_key=prompt', 'data.truncation=left', 'data.max_prompt_length=512', 'data.max_response_length=1536', 'data.gen_batch_size=4', 'data.train_batch_size=4', 'actor_rollout_ref.rollout.n=8', 'algorithm.adv_estimator=grpo', 'algorithm.use_kl_in_reward=False', 'algorithm.kl_ctrl.kl_coef=0.0', 'actor_rollout_ref.actor.use_kl_loss=False', 'actor_rollout_ref.actor.kl_loss_coef=0.0', 'actor_rollout_ref.actor.clip_ratio_low=0.2', 'actor_rollout_ref.actor.clip_ratio_high=0.28', 'actor_rollout_ref.actor.clip_ratio_c=10.0', 'algorithm.filter_groups.enable=True', 'algorithm.filter_groups.metric=seq_final_reward', 'actor_rollout_ref.model.use_remove_padding=False', 'actor_rollout_ref.actor.use_dynamic_bsz=False', 'actor_rollout_ref.ref.log_prob_use_dynamic_bsz=False', 'actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=False', 'actor_rollout_ref.actor.ppo_max_token_len_per_gpu=2048', 'actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=2048', 'actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=2048', 'actor_rollout_ref.model.path=/data01/linxiang/ft_qwen2audio/exp/model_full_v6/qwen2-audio-7b-instruct/v0-20250603-150803/Qwen2-Audio-7b-instruct', 'actor_rollout_ref.model.enable_gradient_checkpointing=True', 'actor_rollout_ref.actor.optim.lr=1e-6', 'actor_rollout_ref.actor.optim.lr_warmup_steps=10', 'actor_rollout_ref.actor.optim.weight_decay=0.1', 'actor_rollout_ref.actor.ppo_mini_batch_size=1', 'actor_rollout_ref.actor.fsdp_config.param_offload=True', 'actor_rollout_ref.actor.fsdp_config.optimizer_offload=True', 'actor_rollout_ref.actor.entropy_coeff=0', 'actor_rollout_ref.actor.grad_clip=1.0', 'actor_rollout_ref.actor.loss_agg_mode=seq-mean-token-mean', 'actor_rollout_ref.actor.ulysses_sequence_parallel_size=1', 'actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1', 'actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1', 'actor_rollout_ref.rollout.gpu_memory_utilization=0.80', 'actor_rollout_ref.rollout.tensor_model_parallel_size=1', 'actor_rollout_ref.rollout.enable_chunked_prefill=True', 'actor_rollout_ref.rollout.max_num_batched_tokens=2048', 'actor_rollout_ref.rollout.temperature=1.0', 'actor_rollout_ref.rollout.top_p=1.0', 'actor_rollout_ref.rollout.top_k=-1', 'actor_rollout_ref.rollout.val_kwargs.temperature=1.0', 'actor_rollout_ref.rollout.val_kwargs.top_p=0.7', 'actor_rollout_ref.rollout.val_kwargs.top_k=-1', 'actor_rollout_ref.rollout.val_kwargs.do_sample=True', 'actor_rollout_ref.rollout.val_kwargs.n=1', 'actor_rollout_ref.ref.fsdp_config.param_offload=True', 'actor_rollout_ref.ref.ulysses_sequence_parallel_size=1', 'actor_rollout_ref.actor.fsdp_config.fsdp_size=-1', 'reward_model.reward_manager=dapo', 'reward_model.overlong_buffer.enable=False', 'reward_model.overlong_buffer.len=4096', 'reward_model.overlong_buffer.penalty_factor=1.0', 'trainer.logger=[console]', 'trainer.project_name=DAPO', 'trainer.experiment_name=DAPO-SFT-base-Our-Merge', 'trainer.n_gpus_per_node=4', 'trainer.nnodes=1', 'trainer.val_before_train=False', 'trainer.test_freq=9999', 'trainer.save_freq=10', 'trainer.total_epochs=1', 'trainer.default_local_dir=/data01/linxiang/DAPO/exp/DAPO-SFT-base-Our-Merge', 'trainer.resume_mode=auto']
Traceback (most recent call last):
  File "/data01/linxiang/verl/recipe/dapo/main_dapo.py", line 29, in main
    run_ppo(config)
  File "/data01/linxiang/verl/recipe/dapo/main_dapo.py", line 41, in run_ppo
    ray.get(runner.run.remote(config))
  File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
  File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/worker.py", line 2849, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/ray/_private/worker.py", line 937, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ActorDiedError): �[36mray::TaskRunner.run()�[39m (pid=493818, ip=10.0.30.111, actor_id=6a5b7ebbcd1c3ca37ee09cca01000000, repr=<main_dapo.TaskRunner object at 0x7f776ffcab60>)
  File "/data01/linxiang/verl/recipe/dapo/main_dapo.py", line 180, in run
    trainer.fit()
  File "/data01/linxiang/verl/recipe/dapo/dapo_ray_trainer.py", line 121, in fit
    gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch)
  File "/data01/linxiang/verl/verl/single_controller/ray/base.py", line 51, in __call__
    output = ray.get(output)
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
	class_name: create_colocated_worker_cls.<locals>.WorkerDict
	actor_id: 7710216a6912f94f21d0dafe01000000
	pid: 494481
	name: u0ROzkWorkerDict_0:0
	namespace: fabeee12-0caa-43c7-884d-63df03d73e8d
	ip: 10.0.30.111
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.

I wonder if its the version conflict?? My pip list is following:

Package                                  Version
---------------------------------------- -------------
accelerate                               1.8.1
aiohappyeyeballs                         2.6.1
aiohttp                                  3.12.13
aiohttp-cors                             0.8.1
aiosignal                                1.3.2
airportsdata                             20250622
annotated-types                          0.7.0
antlr4-python3-runtime                   4.9.3
anyio                                    4.9.0
astor                                    0.8.1
async-timeout                            5.0.1
attrs                                    25.3.0
blake3                                   1.0.5
cachetools                               5.5.2
certifi                                  2025.6.15
cfgv                                     3.4.0
charset-normalizer                       3.4.2
click                                    8.2.1
cloudpickle                              3.1.1
codetiming                               1.4.0
colorful                                 0.5.6
compressed-tensors                       0.9.2
cupy-cuda12x                             13.4.1
datasets                                 3.6.0
Deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.3.8
diskcache                                5.6.3
distlib                                  0.3.9
distro                                   1.9.0
dnspython                                2.7.0
einops                                   0.8.1
email_validator                          2.2.0
exceptiongroup                           1.3.0
fastapi                                  0.115.13
fastapi-cli                              0.0.7
fastrlock                                0.8.3
filelock                                 3.18.0
frozenlist                               1.7.0
fsspec                                   2025.3.0
gguf                                     0.10.0
gitdb                                    4.0.12
GitPython                                3.1.44
google-api-core                          2.25.1
google-auth                              2.40.3
googleapis-common-protos                 1.70.0
grpcio                                   1.73.0
h11                                      0.16.0
hf-xet                                   1.1.5
httpcore                                 1.0.9
httptools                                0.6.4
httpx                                    0.28.1
huggingface-hub                          0.33.0
hydra-core                               1.3.2
identify                                 2.6.12
idna                                     3.10
importlib_metadata                       8.0.0
interegular                              0.3.3
Jinja2                                   3.1.6
jiter                                    0.10.0
jsonschema                               4.24.0
jsonschema-specifications                2025.4.1
lark                                     1.2.2
latex2sympy2_extended                    1.10.2
liger_kernel                             0.5.10
llguidance                               0.7.30
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
markdown-it-py                           3.0.0
MarkupSafe                               3.0.2
math-verify                              0.8.0
mdurl                                    0.1.2
mistral_common                           1.6.2
mpmath                                   1.3.0
msgpack                                  1.1.1
msgspec                                  0.19.0
multidict                                6.5.1
multiprocess                             0.70.16
nanobind                                 2.7.0
nest-asyncio                             1.6.0
networkx                                 3.4.2
ninja                                    1.11.1.4
nodeenv                                  1.9.1
numba                                    0.61.0
numpy                                    2.1.3
nvidia-cublas-cu12                       12.4.5.8
nvidia-cuda-cupti-cu12                   12.4.127
nvidia-cuda-nvrtc-cu12                   12.4.127
nvidia-cuda-runtime-cu12                 12.4.127
nvidia-cudnn-cu12                        9.1.0.70
nvidia-cufft-cu12                        11.2.1.3
nvidia-curand-cu12                       10.3.5.147
nvidia-cusolver-cu12                     11.6.1.9
nvidia-cusparse-cu12                     12.3.1.170
nvidia-cusparselt-cu12                   0.6.2
nvidia-nccl-cu12                         2.21.5
nvidia-nvjitlink-cu12                    12.4.127
nvidia-nvtx-cu12                         12.4.127
omegaconf                                2.3.0
openai                                   1.91.0
opencensus                               0.11.4
opencensus-context                       0.1.3
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-exporter-prometheus        0.47b0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.9
orjson                                   3.10.18
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                25.0
pandas                                   2.3.0
partial-json-parser                      0.2.1.1.post6
peft                                     0.15.2
pillow                                   11.2.1
pip                                      25.1
platformdirs                             4.3.8
pre_commit                               4.2.0
prometheus_client                        0.22.1
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.3.2
proto-plus                               1.26.1
protobuf                                 4.25.8
psutil                                   7.0.0
py-cpuinfo                               9.0.0
py-spy                                   0.4.0
pyarrow                                  19.0.0
pyasn1                                   0.6.1
pyasn1_modules                           0.4.2
pybind11                                 2.13.6
pycountry                                24.6.1
pydantic                                 2.11.7
pydantic_core                            2.33.2
Pygments                                 2.19.2
pylatexenc                               2.10
python-dateutil                          2.9.0.post0
python-dotenv                            1.1.1
python-json-logger                       3.3.0
python-multipart                         0.0.20
pytz                                     2025.2
PyYAML                                   6.0.2
pyzmq                                    27.0.0
ray                                      2.47.1
referencing                              0.36.2
regex                                    2024.11.6
requests                                 2.32.4
rich                                     14.0.0
rich-toolkit                             0.14.7
rpds-py                                  0.25.1
rsa                                      4.9.1
safetensors                              0.5.3
scipy                                    1.15.3
sentencepiece                            0.2.0
sentry-sdk                               2.31.0
setproctitle                             1.3.6
setuptools                               78.1.1
shellingham                              1.5.4
six                                      1.17.0
smart-open                               7.1.0
smmap                                    5.0.2
sniffio                                  1.3.1
starlette                                0.46.2
sympy                                    1.13.1
tabulate                                 0.9.0
tensordict                               0.6.2
tiktoken                                 0.9.0
tokenizers                               0.21.2
torch                                    2.6.0
torchaudio                               2.6.0
torchdata                                0.11.0
torchvision                              0.21.0
tqdm                                     4.67.1
transformers                             4.52.4
triton                                   3.2.0
typer                                    0.16.0
typing_extensions                        4.14.0
typing-inspection                        0.4.1
tzdata                                   2025.2
urllib3                                  2.5.0
uvicorn                                  0.34.3
uvloop                                   0.21.0
verl                                     0.4.1
virtualenv                               20.31.2
vllm                                     0.8.3
wandb                                    0.20.1
watchfiles                               1.1.0
websockets                               15.0.1
wheel                                    0.45.1
wrapt                                    1.17.2
xformers                                 0.0.29.post2
xgrammar                                 0.1.17
xxhash                                   3.5.0
yarl                                     1.20.1
zipp                                     3.23.0

and my command is:

#!/usr/bin/env bash
set -xeuo pipefail

export CUDA_VISIBLE_DEVICES=4,5,6,7
GPU_NUM=4 

project_name='DAPO'

exp_name='DAPO-SFT-base-AVQA'

adv_estimator=grpo

save_freq=10
test_freq=9999

use_kl_in_reward=False
kl_coef=0.0
use_kl_loss=False
kl_loss_coef=0.0

clip_ratio_low=0.2
clip_ratio_high=0.28

max_prompt_length=512 # 2048 1536
max_response_length=1536 # 2536 20480 1536

enable_overlong_buffer=False
overlong_buffer_len=$((512 * 4))
overlong_penalty_factor=1.0

loss_agg_mode="seq-mean-token-mean"

filter_groups_metric="seq_final_reward"  
# max_num_gen_batches=10 
enable_filter_groups=True
gen_prompt_bsz=4 
train_prompt_bsz=4 
train_prompt_mini_bsz=1
n_resp_per_prompt=8 

NNODES=1 

# MODEL_PATH=${MODEL_PATH:-"/Qwen2-Audio-7b-instruct"}
MODEL_PATH=${MODEL_PATH:-"/Qwen2-Audio-7b-instruct"}

CKPTS_DIR=${CKPTS_DIR:-"/DAPO/exp/${exp_name}"}

TRAIN_FILE=${TRAIN_FILE:-"/rain_qa_updated.jsonl"}
TEST_FILE=${TEST_FILE:-"/apo_merge_all_0618_val.jsonl"}

# Algorithm
temperature=1.0
top_p=1.0
top_k=-1 # 0 for HF rollout, -1 for vLLM rollout
val_top_p=0.7

# Performance Related Parameter
sp_size=1
use_dynamic_bsz=False 
actor_ppo_max_token_len=$((max_prompt_length + max_response_length))
infer_ppo_max_token_len=$((max_prompt_length + max_response_length))
offload=True 
gen_tp=1

micro_batch_size=1

export RAY_DISABLE_WORKER_RESTART=1 

export VLLM_USE_V1=1

python3 -m recipe.dapo.main_dapo \
    data.train_files="${TRAIN_FILE}" \
    data.val_files="${TEST_FILE}" \
    data.prompt_key=prompt \
    data.truncation='left' \
    data.max_prompt_length=${max_prompt_length} \
    data.max_response_length=${max_response_length} \
    data.gen_batch_size=${gen_prompt_bsz} \
    data.train_batch_size=${train_prompt_bsz} \
    actor_rollout_ref.rollout.n=${n_resp_per_prompt} \
    algorithm.adv_estimator=${adv_estimator} \
    algorithm.use_kl_in_reward=${use_kl_in_reward} \
    algorithm.kl_ctrl.kl_coef=${kl_coef} \
    actor_rollout_ref.actor.use_kl_loss=${use_kl_loss} \
    actor_rollout_ref.actor.kl_loss_coef=${kl_loss_coef} \
    actor_rollout_ref.actor.clip_ratio_low=${clip_ratio_low} \
    actor_rollout_ref.actor.clip_ratio_high=${clip_ratio_high} \
    actor_rollout_ref.actor.clip_ratio_c=10.0 \
    algorithm.filter_groups.enable=${enable_filter_groups} \
    algorithm.filter_groups.metric=${filter_groups_metric} \
    actor_rollout_ref.model.use_remove_padding=False \
    actor_rollout_ref.actor.use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.ref.log_prob_use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=${use_dynamic_bsz} \
    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=${actor_ppo_max_token_len} \
    actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=${infer_ppo_max_token_len} \
    actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=${infer_ppo_max_token_len} \
    actor_rollout_ref.model.path="${MODEL_PATH}" \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.actor.optim.lr_warmup_steps=10 \
    actor_rollout_ref.actor.optim.weight_decay=0.1 \
    actor_rollout_ref.actor.ppo_mini_batch_size=${train_prompt_mini_bsz} \
    actor_rollout_ref.actor.fsdp_config.param_offload=${offload} \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=${offload} \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.actor.grad_clip=1.0 \
    actor_rollout_ref.actor.loss_agg_mode=${loss_agg_mode} \
    actor_rollout_ref.actor.ulysses_sequence_parallel_size=${sp_size} \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=${micro_batch_size} \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=${micro_batch_size} \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.80 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=${gen_tp} \
    actor_rollout_ref.rollout.enable_chunked_prefill=True \
    actor_rollout_ref.rollout.max_num_batched_tokens=$((max_prompt_length + max_response_length)) \
    actor_rollout_ref.rollout.temperature=${temperature} \
    actor_rollout_ref.rollout.top_p=${top_p} \
    actor_rollout_ref.rollout.top_k="${top_k}" \
    actor_rollout_ref.rollout.val_kwargs.temperature=${temperature} \
    actor_rollout_ref.rollout.val_kwargs.top_p=${val_top_p} \
    actor_rollout_ref.rollout.val_kwargs.top_k=${top_k} \
    actor_rollout_ref.rollout.val_kwargs.do_sample=True \
    actor_rollout_ref.rollout.val_kwargs.n=1 \
    actor_rollout_ref.ref.fsdp_config.param_offload=${offload} \
    actor_rollout_ref.ref.ulysses_sequence_parallel_size=${sp_size} \
    actor_rollout_ref.actor.fsdp_config.fsdp_size=-1 \
    reward_model.reward_manager=dapo \
    reward_model.overlong_buffer.enable=${enable_overlong_buffer} \
    reward_model.overlong_buffer.len=${overlong_buffer_len} \
    reward_model.overlong_buffer.penalty_factor=${overlong_penalty_factor} \
    trainer.logger=['console'] \
    trainer.project_name="${project_name}" \
    trainer.experiment_name="${exp_name}" \
    trainer.n_gpus_per_node=${GPU_NUM} \
    trainer.nnodes=1 \
    trainer.val_before_train=False \
    trainer.test_freq=${test_freq} \
    trainer.save_freq=${save_freq} \
    trainer.total_epochs=1 \
    trainer.default_local_dir="${CKPTS_DIR}" \
    trainer.resume_mode=auto \
    > /data01/linxiang/DAPO/logs/AVQA_0703-8.txt 2>&1 

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ruisearch42]

Hi @Jacoblincc , you got an ActorDiedError, which is likely to be caused by:
Fatal Python error: none_dealloc: deallocating None

This is when sleep() was called:

�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/device_allocator/cumem.py", line 81 in unmap_and_release
�[36m(WorkerDict pid=494481)�[0m   File "/home/linxiang/miniconda3/envs/DAPO/lib/python3.10/site-packages/vllm/device_allocator/cumem.py", line 206 in sleep

You want to look into why None was passed in for deallocation.

[JingbiaoMei]

Similar error, but not training in verl, just using online serving requests:

INFO 08-06 00:36:13 [loggers.py:118] Engine 000: Avg prompt throughput: 2044.3 tokens/s, Avg generation throughput: 567.4 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.9%, Prefix cache hit rate: 18.7%
INFO:     127.0.0.1:57216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:42516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:42534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:57262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Fatal Python error: none_dealloc: deallocating None: bug likely caused by a refcount error in a C extension
Python runtime state: initialized

Thread 0x00007f01db6fe6c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 327 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/queue.py", line 171 in get
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 779 in process_output_sockets
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 982 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007f01dbeff6c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/zmq/sugar/poll.py", line 106 in poll
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 734 in process_input_sockets
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 982 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007eff71dff6c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 331 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 629 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007eff82ffd6c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 331 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 629 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007eff827fc6c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 331 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 629 in wait
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00007eff8bef16c0 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/usage/usage_lib.py", line 233 in _report_continuous_usage
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/usage/usage_lib.py", line 168 in _report_usage_worker
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 982 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x00007f03ff95a600 (most recent call first):
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/kv_cache_utils.py", line 258 in append
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/block_pool.py", line 291 in free_blocks
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 172 in free
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/kv_cache_coordinator.py", line 123 in free
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/kv_cache_manager.py", line 306 in free
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/sched/scheduler.py", line 994 in _free_blocks
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/sched/scheduler.py", line 988 in _free_request
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/core/sched/scheduler.py", line 805 in update_from_output
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 236 in step
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 631 in _process_engine_step
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 606 in run_busy_loop
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 579 in run_engine_core
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/multiprocessing/spawn.py", line 135 in _main
  File "/home/howard/anaconda3/envs/vllm/lib/python3.11/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, msgspec._core, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imaging, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, psutil._psutil_linux, psutil._psutil_posix, zmq.backend.cython._zmq, pybase64._pybase64, sentencepiece._sentencepiece, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, xxhash._xxhash, pyarrow._acero, pyarrow._csv, pyarrow._json, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, uvloop.loop, vllm.cumem_allocator, cuda_utils, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.typing.builtins.itertools, numba.cpython.builtins.math, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, __triton_launcher (total: 167)
INFO 08-06 00:36:23 [loggers.py:118] Engine 000: Avg prompt throughput: 545.7 tokens/s, Avg generation throughput: 111.8 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 17.9%
INFO 08-06 00:36:33 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 17.9%
^CINFO 08-06 11:20:00 [launcher.py:80] Shutting down FastAPI HTTP server.

My environment is wsl ubuntu2404lts and

Package                           Version
--------------------------------- -------------
aiohappyeyeballs                  2.6.1
aiohttp                           3.12.15
aiosignal                         1.4.0
airportsdata                      20250706
annotated-types                   0.7.0
anyio                             4.10.0
astor                             0.8.1
attrs                             25.3.0
blake3                            1.0.5
cachetools                        6.1.0
certifi                           2025.8.3
charset-normalizer                3.4.2
click                             8.2.1
cloudpickle                       3.1.1
compressed-tensors                0.10.2
cuda-bindings                     12.9.0
cuda-python                       12.9.0
cupy-cuda12x                      13.5.1
datasets                          4.0.0
depyf                             0.18.0
dill                              0.3.8
diskcache                         5.6.3
distro                            1.9.0
dnspython                         2.7.0
einops                            0.8.1
email_validator                   2.2.0
fastapi                           0.116.1
fastapi-cli                       0.0.8
fastapi-cloud-cli                 0.1.5
fastrlock                         0.8.3
filelock                          3.18.0
flash_attn                        2.8.2
frozenlist                        1.7.0
fsspec                            2025.3.0
gguf                              0.17.1
h11                               0.16.0
hf-xet                            1.1.5
httpcore                          1.0.9
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.34.3
idna                              3.10
interegular                       0.3.3
Jinja2                            3.1.6
jiter                             0.10.0
jsonschema                        4.25.0
jsonschema-specifications         2025.4.1
lark                              1.2.2
llguidance                        0.7.30
llvmlite                          0.44.0
lm-format-enforcer                0.10.11
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
mdurl                             0.1.2
mistral_common                    1.8.3
mpmath                            1.3.0
msgpack                           1.1.1
msgspec                           0.19.0
multidict                         6.6.3
multiprocess                      0.70.16
nest-asyncio                      1.6.0
networkx                          3.5
ninja                             1.11.1.4
numba                             0.61.2
numpy                             2.2.6
nvidia-cublas-cu12                12.8.3.14
nvidia-cuda-cupti-cu12            12.8.57
nvidia-cuda-nvrtc-cu12            12.8.61
nvidia-cuda-runtime-cu12          12.8.57
nvidia-cudnn-cu12                 9.7.1.26
nvidia-cufft-cu12                 11.3.3.41
nvidia-cufile-cu12                1.13.0.11
nvidia-curand-cu12                10.3.9.55
nvidia-cusolver-cu12              11.7.2.55
nvidia-cusparse-cu12              12.5.7.53
nvidia-cusparselt-cu12            0.6.3
nvidia-ml-py                      12.575.51
nvidia-nccl-cu12                  2.26.2
nvidia-nvjitlink-cu12             12.8.61
nvidia-nvshmem-cu12               3.3.20
nvidia-nvtx-cu12                  12.8.55
openai                            1.90.0
opencv-python-headless            4.12.0.88
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         25.0
pandas                            2.3.1
partial-json-parser               0.2.1.1.post6
pillow                            11.3.0
pip                               25.1
prometheus_client                 0.22.1
prometheus-fastapi-instrumentator 7.1.0
propcache                         0.3.2
protobuf                          6.31.1
psutil                            7.0.0
py-cpuinfo                        9.0.0
pyarrow                           21.0.0
pybase64                          1.4.2
pycountry                         24.6.1
pydantic                          2.11.7
pydantic_core                     2.33.2
pydantic-extra-types              2.10.5
Pygments                          2.19.2
pynvml                            12.0.0
python-dateutil                   2.9.0.post0
python-dotenv                     1.1.1
python-json-logger                3.3.0
python-multipart                  0.0.20
pytz                              2025.2
PyYAML                            6.0.2
pyzmq                             27.0.1
ray                               2.48.0
referencing                       0.36.2
regex                             2025.7.34
requests                          2.32.4
rich                              14.1.0
rich-toolkit                      0.14.9
rignore                           0.6.4
rpds-py                           0.26.0
safetensors                       0.5.3
scipy                             1.16.1
sentencepiece                     0.2.0
sentry-sdk                        2.34.1
setuptools                        78.1.1
shellingham                       1.5.4
six                               1.17.0
sniffio                           1.3.1
starlette                         0.47.2
sympy                             1.14.0
tiktoken                          0.9.0
tokenizers                        0.21.4
torch                             2.7.0+cu128
torchaudio                        2.7.0+cu128
torchvision                       0.22.0+cu128
tqdm                              4.67.1
transformers                      4.53.3
triton                            3.3.0
typer                             0.16.0
typing_extensions                 4.14.1
typing-inspection                 0.4.1
tzdata                            2025.2
urllib3                           2.5.0
uv                                0.8.4
uvicorn                           0.35.0
uvloop                            0.21.0
vllm                              0.9.2
watchfiles                        1.1.0
websockets                        15.0.1
wheel                             0.45.1
xformers                          0.0.30
xgrammar                          0.1.19
xxhash                            3.5.0
yarl                              1.20.1

[ruisearch42]

Hi @JingbiaoMei , to clarify, were you using ray in your test? And how to reproduce the error?

[Ladazhx]

我遇到了跟 @JingbiaoMei 几乎完全一样的错误，请问这个问题现在有定位到是什么原因？有什么具体的解决办法吗？

[ruisearch42]

Hi @Ladazhx we need more info to talk about possibilities and look further, can you respond to the same question:

to clarify, were you using ray in your test? And how to reproduce the error?

[Ladazhx]

@ruisearch42 Sorry for replying so late. I'll try his method first. If it doesn't work, we can then discuss further.

Hi @Ladazhx we need more info to talk about possibilities and look further, can you respond to the same question:

to clarify, were you using ray in your test? And how to reproduce the error?

Sorry for replying so late. I'll try this branch first. If it doesn't work, we can then discuss further.

[JingbiaoMei]

Hi @JingbiaoMei , to clarify, were you using ray in your test? And how to reproduce the error?

Hi,

Sorry for the late reply. I was not using ray.

To reproduce this error, I run
vllm serve ….

And fire up 100 requests continuously, i.e. fire another 100 requests after the first batch completes, the error occurs after around 30min-2hr of continuous request.

[youkaichao]

should be fixed by #23477