Don't return cached heartbeat read when query service is down to avoi… #4689
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jschlather
force-pushed
the
heartbeat-fix
branch
from
1745509
to
55d7e3d
Compare
leoxlin
reviewed
Feb 28, 2019
| // We return healthy from this as a signal to the healtcheck to attempt | ||
| // to start the query service again | ||
| if !tsv.hr.IsOpen() && !tsv.IsServing() { | ||
| return 0, nil |
There was a problem hiding this comment.
Is it possible to return something to indicate the reader is closed?
i.e. -1 or max int
There was a problem hiding this comment.
We could return -1, if we return max int, then we will get an unhealthy due to replication lag issue. We could also return a custom error that specifies the reader is closed. We would need to add code to handle -1 or the custom error inside the healthcheck.
There was a problem hiding this comment.
Given that this is an improvement already, I'll merge this. We can fix forward as needed.
…d situation where replica tablet never recovers. Signed-off-by: Jacob Schlather <jschlather@hubspot.com> Signed-off-by: <jschlather@hubspot.com> Signed-off-by: <jschlather@hubspot.com>
jschlather
force-pushed
the
heartbeat-fix
branch
from
55d7e3d
to
0d01465
Compare
|
@sougou Could you take a look? Thank you. |
sougou
approved these changes
Mar 12, 2019
| // We return healthy from this as a signal to the healtcheck to attempt | ||
| // to start the query service again | ||
| if !tsv.hr.IsOpen() && !tsv.IsServing() { | ||
| return 0, nil |
There was a problem hiding this comment.
Given that this is an improvement already, I'll merge this. We can fix forward as needed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…d situation where replica tablet never recovers.
Closes #4673
The core of the problem in 4673, is that the HeartbeatReporter does not obey the contract expected by the healthcheck - that it reports a current reading of the health. I don't see a straightforward way to fix this without rewriting the reader entirely. The next best thing I could see to do was to return healthy when we knew the reader was closed and the query service was shutdown. This will force the healthcheck to attempt to restart the query service and if that fails, then it will report that error and remain unhealthy. I considered putting this logic in the reader, but if we only know the reader is closed then there's the possibility of a pathological situation where the query service is running and the reader is closed and then we would have no visibility into this.
cc @leoxlin