[core] Fix gcs healthch manager crash when node is removed by node manager. #31917
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yi Cheng <74173148+iycheng@users.noreply.github.com>
fishbone
commented
Jan 25, 2023
| @@ -30,6 +34,20 @@ using namespace boost; | |||
| #include "gtest/gtest.h" | |||
| #include "ray/gcs/gcs_server/gcs_health_check_manager.h" | |||
|
|
|||
| int GetFreePort() { | |||
There was a problem hiding this comment.
fun fact: it's generated by chatgpt
fishbone
changed the title
|
still failed in testing env |
|
Signed-off-by: Yi Cheng <74173148+iycheng@users.noreply.github.com>
scv119
reviewed
Jan 26, 2023
| @@ -121,14 +110,14 @@ class GcsHealthCheckManager { | |||
| NodeID node_id_; | |||
|
|
|||
| // Whether the health check has stopped. | |||
| std::shared_ptr<bool> stopped_; | |||
| bool stopped_ = false; | |||
|
|
|||
| /// gRPC related fields | |||
| std::unique_ptr<::grpc::health::v1::Health::Stub> stub_; | |||
|
|
|||
| // The context is used in the gRPC callback which is in another | |||
scv119
reviewed
Jan 26, 2023
| context_ = std::make_shared<grpc::ClientContext>(); | ||
| // Reset the context/request/response for the next request. | ||
| context_.~ClientContext(); | ||
| new (&context_) grpc::ClientContext(); |
There was a problem hiding this comment.
if we make context_ an pointer or unique_ptr, would it make it more nature?
There was a problem hiding this comment.
It actually doesn't matter any difference I think since at the same time, only one in-flight request will be there per node.
I did this just following what gRPC did (https://sourcegraph.com/github.com/grpc/grpc/-/blob/test/cpp/qps/client_sync.cc?L189:26) and I think this is for performance (allocate on stack vs allocate on heap)(perf is not very important here, so I think no big difference.)
scv119
approved these changes
Jan 26, 2023
scv119
added
the
@author-action-required
edoakes
pushed a commit
to edoakes/ray
that referenced
this pull request
Mar 22, 2023
… node manager. (ray-project#31917)" (ray-project#31995) This reverts commit a32b9b1. Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The root cause is because the data structure is deleted, but call backs is not canceled and got executed. This PR simplify the life model and make it the way gRPC works. We only delete the structure after gRPC OnDone is called.
In the shortcut, according to the doc https://github.com/grpc/proposal/blob/master/L67-cpp-callback-api.md#unary-rpc-shortcuts , OnDone will call the callback function.
A better model is needed here. The code will be changed once we update the threading model in gRPC.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.