Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race when stopping chunks memcached client #4511

Merged
merged 2 commits into from
Oct 14, 2021

Conversation

bboreham
Copy link
Contributor

@bboreham bboreham commented Oct 1, 2021

We need to check again if we have been asked to quit before writing to the results chan.

Also, the reader of that chan should not close it, because this can trigger a panic in a writer. The chan will be cleaned up by garbage-collection once all the readers and writers have exited on quit.

Which issue(s) this PR fixes:
Fixes #4508

Checklist

  • NA Tests updated
  • NA Documentation added
  • NA CHANGELOG.md updated

We need to check again if we have been asked to quit before writing to
the results chan.

Also, the reader of that chan should not close it, because this can
trigger a panic in a writer. The chan will be cleaned up by garbage-
collection once all the readers and writers have exited on quit.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
case <-c.quit:
return
default:
input.resultCh <- res
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sending on the channel inside the default case seems that it could still block if it enters the default clause just before quit is closed.
Can we make it a case instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry was on vacation. Very good point; done.

Don't open another race where quit is closed just after we check it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
@rsteneteg
Copy link
Contributor

LGTM

@alvinlin123 alvinlin123 merged commit e9702b8 into master Oct 14, 2021
srijan55 pushed a commit to srijan55/cortex that referenced this pull request Nov 26, 2021
* Fix race when stopping chunks memcached client

We need to check again if we have been asked to quit before writing to
the results chan.

Also, the reader of that chan should not close it, because this can
trigger a panic in a writer. The chan will be cleaned up by garbage-
collection once all the readers and writers have exited on quit.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* simplify select

Don't open another race where quit is closed just after we check it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Manish Kumar Gupta <manishkg@microsoft.com>
alvinlin123 pushed a commit to ac1214/cortex that referenced this pull request Jan 14, 2022
* Fix race when stopping chunks memcached client

We need to check again if we have been asked to quit before writing to
the results chan.

Also, the reader of that chan should not close it, because this can
trigger a panic in a writer. The chan will be cleaned up by garbage-
collection once all the readers and writers have exited on quit.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* simplify select

Don't open another race where quit is closed just after we check it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Alvin Lin <alvinlin@amazon.com>
@friedrichg friedrichg deleted the fix-chunk-cache-race branch February 15, 2024 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Data race in TestMemcacheStop
3 participants