Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI test linux://:mutable_object_test is flaky #44396

Closed
can-anyscale opened this issue Apr 1, 2024 · 22 comments · Fixed by #44413, #44720 or #44764
Closed

CI test linux://:mutable_object_test is flaky #44396

can-anyscale opened this issue Apr 1, 2024 · 22 comments · Fixed by #44413, #44720 or #44764
Assignees
@jackhumphries
Labels
bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core core-object-store flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ P0 Issues that should be fixed in short order ray-test-bot Issues managed by OSS test policy stability

Comments

@can-anyscale
Copy link
Collaborator

CI test linux://:mutable_object_test is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/3797#018e9b5a-03dd-4967-a20f-45116141500e
- https://buildkite.com/ray-project/postmerge/builds/3795#018e9b0a-9d1d-445b-b5c2-01143ba361f1
- https://buildkite.com/ray-project/postmerge/builds/3791#018e9a7b-5bde-4252-b036-8f11e8282f5d
- https://buildkite.com/ray-project/postmerge/builds/3792#018e9a81-897d-4bd3-9df5-c100f599a5ef
- https://buildkite.com/ray-project/postmerge/builds/3791#018e9a7b-5bda-42bb-9e4e-f167c7fc54a5

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy
@can-anyscale can-anyscale added bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ ray-test-bot Issues managed by OSS test policy stability triage Needs triage (eg: priority, bug/not-bug, and owning component) weekly-release-blocker Issues that will be blocking Ray weekly releases labels Apr 1, 2024
@can-anyscale
Copy link
Collaborator Author

Image

@can-anyscale
Copy link
Collaborator Author

I'm not sure whether it's because of that PR but @jackhumphries is the master mind behind cpp anyway ;)

@jjyao jjyao removed their assignment Apr 1, 2024
@jjyao jjyao added P0 Issues that should be fixed in short order and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Apr 1, 2024
jackhumphries added a commit to jackhumphries/ray that referenced this issue Apr 2, 2024
@jackhumphries
Fixes the flaky mutable_object_test
b4460fe 
Based on a local issue I resolved, I believe this commit should fix the
flaky mutable_object_test in ray-project#44396.

The header of the mutable object was not being explicitly initialized.
In some cases, this caused deadlock due to a spinlock backed by garbage memory.

This commit explicitly initializes the header, which should resolve the
issue.

Tested: mutable_object_test

Signed-off-by: Jack Humphries <1645405+jackhumphries@users.noreply.github.com>
@jackhumphries jackhumphries mentioned this issue Apr 2, 2024
Merged
8 tasks
stephanie-wang pushed a commit that referenced this issue Apr 2, 2024
@jackhumphries
Fixes the flaky mutable_object_test (#44413)
1dbcacb 
Based on a local issue I resolved, I believe this commit should fix the
flaky mutable_object_test in #44396.

The header of the mutable object was not being explicitly initialized.
In some cases, this caused deadlock due to a spinlock backed by garbage memory.

This commit explicitly initializes the header, which should resolve the
issue.

Tested: mutable_object_test

Signed-off-by: Jack Humphries <1645405+jackhumphries@users.noreply.github.com>
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/3826#018ea456-8ee6-4b39-9a95-ef990a519880

@jjyao
Copy link
Collaborator

jjyao commented Apr 8, 2024

Still flaky.

@jjyao jjyao reopened this Apr 8, 2024
@jjyao jjyao removed the weekly-release-blocker Issues that will be blocking Ray weekly releases label Apr 8, 2024
@anyscalesam anyscalesam reopened this Apr 8, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/3913#018ec3d8-3c63-4f0d-bcba-77a0fb0a220d
- https://buildkite.com/ray-project/postmerge/builds/3908#018ec24b-fd56-4ea1-881f-7bbfaa5538b6
- https://buildkite.com/ray-project/postmerge/builds/3907#018ec144-7665-4469-9e75-72520d13b2cc
- https://buildkite.com/ray-project/postmerge/builds/3897#018ebe98-b2b1-48e9-8f41-fe6ba0dada6e
- https://buildkite.com/ray-project/postmerge/builds/3891#018eb723-8ce2-4fee-82d3-e21a17b5ceb2

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/3943#018ec9e9-e96f-4739-ab74-a33168c56721

@can-anyscale can-anyscale reopened this Apr 10, 2024
@jackhumphries
Copy link
Contributor

From initial investigation, the test is "flaky" because it takes a long time to run. In ReadAcquire() in src/ray/object_manager/common.cc, there is a polling loop that all readers enter as they wait on the writer to update the object. In each loop, the readers increment and decrement the semaphore (which has a maximum value of 1), so there is significant contention on the semaphore.

@jackhumphries
Copy link
Contributor

jackhumphries commented Apr 13, 2024 •
edited
Loading

A quick fix to reduce contention would be to add sched_yield() between sem_post() and TryToAcquireSemaphore(). A good long term fix would be to use a futex to sleep on the version number to avoid polling.

@stephanie-wang
Copy link
Contributor

stephanie-wang commented Apr 13, 2024 via email

@jackhumphries
Copy link
Contributor

I'd be inclined to keep it as is, because this test should run quickly. It's just unnecessary contention that shouldn't be there in the first place due to polling.

@stephanie-wang
Copy link
Contributor

stephanie-wang commented Apr 13, 2024 via email

@jackhumphries jackhumphries mentioned this issue Apr 13, 2024
Merged
8 tasks
@can-anyscale can-anyscale changed the title CI test linux://:mutable_object_test is flaky CI test linux://:mutable_object_test is consistently_failing Apr 13, 2024
@can-anyscale can-anyscale reopened this Apr 13, 2024
@can-anyscale can-anyscale changed the title CI test linux://:mutable_object_test is flaky CI test linux://:mutable_object_test is consistently_failing Apr 13, 2024
@can-anyscale can-anyscale reopened this Apr 13, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-837e-404e-8f2f-41358e50c2a2
- https://buildkite.com/ray-project/postmerge/builds/4002#018ed621-787a-4b89-8f93-675aac48b2b5
- https://buildkite.com/ray-project/postmerge/builds/4000#018ed611-63b8-43ac-b1dc-1ed38f6e46a0
- https://buildkite.com/ray-project/postmerge/builds/3946#018eca00-78d4-468c-83ee-cea39e555ae9

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

2 similar comments
@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-837e-404e-8f2f-41358e50c2a2
- https://buildkite.com/ray-project/postmerge/builds/4002#018ed621-787a-4b89-8f93-675aac48b2b5
- https://buildkite.com/ray-project/postmerge/builds/4000#018ed611-63b8-43ac-b1dc-1ed38f6e46a0
- https://buildkite.com/ray-project/postmerge/builds/3946#018eca00-78d4-468c-83ee-cea39e555ae9

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-837e-404e-8f2f-41358e50c2a2
- https://buildkite.com/ray-project/postmerge/builds/4002#018ed621-787a-4b89-8f93-675aac48b2b5
- https://buildkite.com/ray-project/postmerge/builds/4000#018ed611-63b8-43ac-b1dc-1ed38f6e46a0
- https://buildkite.com/ray-project/postmerge/builds/3946#018eca00-78d4-468c-83ee-cea39e555ae9

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

@can-anyscale can-anyscale changed the title CI test linux://:mutable_object_test is flaky CI test linux://:mutable_object_test is consistently_failing Apr 13, 2024
@can-anyscale can-anyscale reopened this Apr 13, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-8383-4ee6-a050-2a3bc9d23cc9

1 similar comment
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-8383-4ee6-a050-2a3bc9d23cc9

@can-anyscale can-anyscale changed the title CI test linux://:mutable_object_test is consistently_failing CI test linux://:mutable_object_test is flaky Apr 13, 2024
@can-anyscale can-anyscale reopened this Apr 13, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is flaky. Recent failures:

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/3268#018e0b9c-f120-4741-a4a0-745efa45d938

@can-anyscale can-anyscale reopened this Apr 16, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://:mutable_object_test is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4020#018ee433-c8be-4c70-a12e-72bd591d9956
- https://buildkite.com/ray-project/postmerge/builds/4017#018ee3dc-05e9-41cd-bfe7-91be1b63dff6
- https://buildkite.com/ray-project/postmerge/builds/4014#018ee391-ecc6-40f6-8e6a-7e43cc32c157
- https://buildkite.com/ray-project/postmerge/builds/4004#018edc21-075a-4a9f-81d1-9cb6d7742440
- https://buildkite.com/ray-project/postmerge/builds/4001#018ed618-8381-4ea0-a0dc-96ec70a4402a

DataCaseName-linux://:mutable_object_test-END
Managed by OSS Test Policy

@jackhumphries jackhumphries mentioned this issue Apr 16, 2024
Merged
8 tasks
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4037#018ee8c7-e7ed-4143-8548-a8ee2261c2a4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jackhumphries jackhumphries

Labels
bug Something that is supposed to be working; but isn't ci-test core Issues that should be addressed in Ray Core core-object-store flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ P0 Issues that should be fixed in short order ray-test-bot Issues managed by OSS test policy stability
Projects
None yet
Milestone
No milestone
5 participants
@jjyao @jackhumphries @stephanie-wang @anyscalesam @can-anyscale