Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deflake TestActivityLog_MultipleFragmentsAndSegments #20930

Merged

Conversation

ncabatoff
Copy link
Collaborator

@ncabatoff ncabatoff commented Jun 1, 2023

Example failure: https://github.com/hashicorp/vault-enterprise/actions/runs/5144577938/jobs/9261153205

Fix is never starting the fragment worker instead of racing on stopping it. I added a log (since removed) so we can see more details about failures:

2023-06-01T09:41:46.769-0400 [TRACE] TestActivityLog_MultipleFragmentsAndSegments.activity: new local fragment created
2023-06-01T09:41:46.769-0400 [TRACE] TestActivityLog_MultipleFragmentsAndSegments.activity: shutting down activeFragmentWorkder

The issue was that by closing doneCh, we were hoping that our read of newFragmentCh wouldn't race with activeFragmentWorker's read. However, we don't control when activeFragmentWorker gets scheduled by the runtime, and sometimes it doesn't get scheduled until the new fragment has been created and newFragmentCh is readable. In that case, there's no guarantee that the doneCh read will happen instead of the newFragmentCh read, and when that happens we get

    activity_log_test.go:762: timed out waiting for new fragment

…g the fragment worker instead of racing on stopping it.
@ncabatoff ncabatoff merged commit dc5dd71 into main Jun 2, 2023
@ncabatoff ncabatoff deleted the deflake-TestActivityLog_MultipleFragmentsAndSegments branch June 2, 2023 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants