[Core] Check failed: assigned_job_id_ == job_id Job_id mismatch

[jjyao]

What happened + What you expected to happen

It shouldn't check fail

Versions / Dependencies

master

Reproduction script

ray job submit -- python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --enable-chunked-prefill --max-num-batched-tokens 2048 --max-num-seqs 64 --tokenizer-pool-size 2 --trust-remote-code --tensor-parallel-size 1 --max-model-len 8192

Issue Severity

None

[rynewang]

not reproducible on 2.38

[zhangsikai123]

I encountered the same issue
env: ray 3.10.13, ray 2.42.1 using ray workflow
error log is shown below: [note that 06000000 is the current job that is failed, 02000000 is the job that just finished right before]

^[[33m(raylet)^[[0m [2025-04-20 12:03:44,360 C 86 86] worker.cc:202: Check failed: assigned_job_id_ == job_id Job_id mismatch, assigned: 06000000, actual: 02000000
^[[33m(raylet)^[[0m *** StackTrace Information ***
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xd8965a) [0x56226a93a65a] ray::operator<<()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xd8ba91) [0x56226a93ca91] ray::RayLog::~RayLog()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x35741f) [0x562269f0841f] ray::raylet::Worker::SetJobId()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x358423) [0x562269f09423] ray::raylet::Worker::SetAssignedTask()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x3a8936) [0x562269f59936] ray::raylet::LocalTaskManager::Dispatch()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x3afea8) [0x562269f60ea8] ray::raylet::LocalTaskManager::PoppedWorkerHandler()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x3b0394) [0x562269f61394] std::_Function_handler<>::_M_invoke()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x337b36) [0x562269ee8b36] ray::raylet::WorkerPool::PopWorker()::{lambda()#1}::operator()()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x34c0e4) [0x562269efd0e4] ray::raylet::WorkerPool::PushWorker()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x32ee80) [0x562269edfe80] ray::raylet::NodeManager::HandleWorkerAvailable()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x32f36a) [0x562269ee036a] ray::raylet::NodeManager::ProcessAnnounceWorkerPortMessageImpl()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x331d22) [0x562269ee2d22] ray::raylet::NodeManager::ProcessClientMessage()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x351c91) [0x562269f02c91] std::_Function_handler<>::_M_invoke()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x689c6d) [0x56226a23ac6d] ray::ClientConnection::ProcessMessage()
^[[33m(raylet)^[[0m /opt/miniconda3/envs/xtrimoengine/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x6a0d28) [0x56226a251d28] EventTracker::RecordExecution()

[zhangsikai123]

another issue posted which I suspect might relate: #52090

[jjyao]

@zhangsikai123 are you using detached actor? Should be fixed by #52405

[zhangsikai123]

We are using ray workflow, the workflowManagementActor is by its definition a detached actor. BTW I saw that fix, but can not use it since it's not in any released version now.