[Core] Fix RAY_CHECK(inserted) inside reference counter #58092
Conversation
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request addresses a RAY_CHECK failure within the reference counter, which occurred due to an incorrect assumption about object ID insertions during task retries with network failures. The fix correctly removes the assertion, allowing for idempotent insertions into the stored_in_objects map. A new regression test has been added to specifically simulate this network failure scenario and validate the fix. The changes are well-targeted, the fix is correct, and the accompanying test ensures this issue will not regress. The removal of an unused import is also a good cleanup. Overall, this is a solid contribution.
edoakes
left a comment
edoakes
left a comment
There was a problem hiding this comment.
Assuming you also audited the rest of the codepath for idempotence?
We should add C++ tests for idempotence of the relevant ref counter methods (following @Sparks0219's playbook)
I checked this particular borrow case but I haven't checked all possible borrow cases yet and I won't be surprised that there might be other issues there. |
…58092) Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: xgui <xgui@anyscale.com>
…58092) Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
…58092) Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>
3 participants
Description
Symptom
reference_counter.cc:1269: An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: inserted
Root cause
The ray check assumption is not right: the
object_idcan be inserted tostored_in_objectsmultiple times in the case of task retry due to network failures. It's fine sincestored_in_objectsis a map.Related issues
Closes #57997
Additional information