Release test llm_batch_vllm failed

[can-anyscale]

Release test llm_batch_vllm failed. See https://buildkite.com/ray-project/release/builds/41101#0196b3dd-3277-4fac-a4d1-6c953b8148f2 for more details.

Managed by OSS Test Policy

[can-anyscale]

Test has been failing for far too long. Jailing.

[lk-chen]

Triggered release test twice on #52229 and both succeeded buildkite1 buildkite2 so I assume this is real test pass. This commit is so far the last commit in ray/llm/ .

The commit after that is #52873 which is just doc update, and it failed release test

---- May 10 update ----

sweeping more parallelism configs 3098a14

=========================== short test summary info ============================
FAILED test_batch_vllm.py::test_vllm_llama_parallel[1-1-2-False] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=4797, ip=10.0.241.84, actor_id=b81a78bb86791ae46418a0f702000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_llama_parallel[2-1-2-True] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=12997, ip=10.0.241.84, actor_id=29e5d68d335626d520b5440202000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_llama_parallel[1-2-2-False] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=14257, ip=10.0.241.84, actor_id=46dbed958eeb76e443bdae7102000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_llama_parallel[2-1-1-True] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=14987, ip=10.0.241.84, actor_id=d7904e4f6f7657f486084b8502000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_llama_parallel[1-2-1-False] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=16052, ip=10.0.241.84, actor_id=8dc152a41dc38cbdf2eb5d2602000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_llama_lora - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, ray::_MapWorker.__init__() (pid=17066, ip=10.0.241.84, actor_id=aed2eb45f39ad055049fd32102000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_vision_language_models[True-llava-hf/llava-1.5-7b-hf-1-1-1-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[True-Qwen/Qwen2.5-VL-3B-Instruct-2-1-2-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[True-Qwen/Qwen2.5-VL-3B-Instruct-1-1-2-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[True-Qwen/Qwen2.5-VL-3B-Instruct-1-1-1-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[False-llava-hf/llava-1.5-7b-hf-1-1-1-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[False-Qwen/Qwen2.5-VL-3B-Instruct-2-1-2-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[False-Qwen/Qwen2.5-VL-3B-Instruct-1-1-2-60]
FAILED test_batch_vllm.py::test_vllm_vision_language_models[False-Qwen/Qwen2.5-VL-3B-Instruct-1-1-1-60]
============ 14 failed, 2 passed, 11 warnings in 942.23s (0:15:42) =============

According to this log

test_batch_vllm.py::test_chat_template_with_vllm 2025-05-11 00:20:59,956	INFO worker.py:1554 -- Using address 10.0.36.198:6379 set in the environment variable RAY_ADDRESS
...
�[A�[A�[A�[A�[A�[A�[36m(_MapWorker pid=3318, ip=10.0.151.59)�[0m �[1;36m(VllmWorker rank=0 pid=4097)�[0;0m INFO 05-11 00:23:21 [monitor.py:33] torch.compile takes 20.26 s in total
...
test_batch_vllm.py::test_vllm_llama_parallel[1-1-2-True] 2025-05-11 00:23:43,836	INFO worker.py:1554 -- Using address 10.0.36.198:6379 set in the environment variable RAY_ADDRESS
...
�[A�[A�[A�[A�[36m(_MapWorker pid=10368, ip=10.0.151.59)�[0m ERROR 05-11 00:24:19 [core.py:396] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.95 GiB of which 9.00 MiB is free. Process 4097 has 20.06 GiB memory in use. Including non-PyTorch memory, this process has 1.87 GiB memory in use. Of the allocated memory 1.66 GiB is allocated by PyTorch, and 4.86 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

So there's some leak during shut down, but #52344 release test is green

[lk-chen]

May 11 update

Trying to isolate potential autoscaler issue from the test in 179c198 log

=========================== short test summary info ============================
FAILED test_batch_vllm.py::test_vllm_llama_parallel[2-1-1-True] - ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, �[36mray::_MapWorker.__init__()�[39m (pid=18033, ip=10.0.222.208, actor_id=580370cf37ce54787f054ce102000000, repr=MapWorker(MapBatches(vLLMEngineStageUDF)))
FAILED test_batch_vllm.py::test_vllm_vision_language_models[False-Qwen/Qwen2.5-VL-3B-Instruct-1-1-1-60]

For test_vllm_vision_language_models[False-Qwen/Qwen2.5-VL-3B-Instruct-1-1-1-60], it failed due to

(_MapWorker pid=3815, ip=10.0.199.225)�[0m AttributeError: 'MultiprocExecutor' object has no attribute 'workers'

which can be fixed in vllm-project/vllm#17811

For the llama_parallel failure

test_batch_vllm.py::test_chat_template_with_vllm 2025-05-11 17:23:57,161	INFO worker.py:1554 -- Using address 10.0.21.19:6379 set in the environment variable RAY_ADDRESS
(_MapWorker pid=3403, ip=10.0.222.208)�[0m INFO 05-11 17:25:15 [core.py:58] Initializing a V1 LLM engine (v0.8.5.post1) with config: model='unsloth/Llama-3.2-1B-Instruct',  ...
(_MapWorker pid=3403, ip=10.0.222.208)�[0m �[1;36m(VllmWorker rank=0 pid=4192)�[0;0m INFO 05-11 17:25:20 [shm_broadcast.py:266] vLLM message queue communication handle: ...
...
test_batch_vllm.py::test_vllm_llama_parallel[2-1-1-True] 2025-05-11 17:27:16,944	INFO worker.py:1554 -- Using address 10.0.21.19:6379 set in the environment variable RAY_ADDRESS
(_MapWorker pid=18033, ip=10.0.222.208)�[0m INFO 05-11 17:27:42 [core.py:58] Initializing a V1 LLM engine (v0.8.5.post1) with config: model='unsloth/Llama-3.2-1B-Instruct', speculative_config=None, ...
(_MapWorker pid=18033, ip=10.0.222.208)�[0m torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 21.95 GiB of which 143.00 MiB is free. Process 4192 has 20.06 GiB memory in use...

So chat_template and parallel[2-1-1-True] tests are scheduled on the same node, and hit the leak issue. I don't get why tests are still scheduled on same node, considering there are a total of 10 test cases, and max of autoscaler is already 10.

[aslonnie]

any new update on this issue? thanks.