Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Fix the GCS memory usage high issue. #32302

Merged
merged 1 commit into from
Feb 8, 2023

Conversation

fishbone
Copy link
Contributor

@fishbone fishbone commented Feb 8, 2023

Why are these changes needed?

It's not because of leak. The root cause is because we allocate more requests when start. This PR fixed it by making the number of call constant.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@fishbone fishbone linked an issue Feb 8, 2023 that may be closed by this pull request
@fishbone fishbone added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order v2.3.0-pick labels Feb 8, 2023
@@ -148,7 +148,7 @@ void GrpcServer::Run() {
if (entry->GetMaxActiveRPCs() != -1) {
buffer_size = entry->GetMaxActiveRPCs();
}
for (int j = 0; j < buffer_size; j++) {
for (int j = 0; j < (buffer_size / num_threads_); j++) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will num_threads_ ever be 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed this one. But I don't think it's going to be a big issue here since it should be always greater than 0.
But yeah, we should enforce it by adding CHECK.

@fishbone fishbone merged commit cf1bc83 into ray-project:master Feb 8, 2023
fishbone added a commit that referenced this pull request Feb 8, 2023
This is a corner case where buffer could be 0 and a comments needs to be fixed in the previous PR.
fishbone added a commit to fishbone/ray that referenced this pull request Feb 8, 2023
…ct#32323)

This is a corner case where buffer could be 0 and a comments needs to be fixed in the previous PR.
scv119 pushed a commit that referenced this pull request Feb 8, 2023
#32330)

This cherry-picks #32302

Why are these changes needed?
It's not because of leak. The root cause is because we allocate more requests when start. This PR fixed it by making the number of call constant.
edoakes pushed a commit to edoakes/ray that referenced this pull request Mar 22, 2023
…ct#32323)

This is a corner case where buffer could be 0 and a comments needs to be fixed in the previous PR.

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core] Likely regression in GCS memory usage from 2.2 to 2.3 releases
3 participants