backupccl: avoid div-by-zero crash on failed node count #55560
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
pbardea
reviewed
Oct 15, 2020
| @@ -1169,7 +1170,11 @@ func (r *restoreResumer) Resume( | |||
|
|
|||
| numClusterNodes, err := clusterNodeCount(p.ExecCfg().Gossip) | |||
There was a problem hiding this comment.
Should we also move this call to closer to the telemetry since that's the only thing that it's used for?
There was a problem hiding this comment.
I sorta considered that but then we'd need to pass Gossip into restore too. I think the real answer is that we should be using number of nodes we actually plan the flow on, so I think this should eventually go away rather than move. In the interest of keeping this backport-friendly I think just leave it as is until then?
We've seen a report of a node that crashed due to a divide-by-zero hit during metrics collection, specifically when computing the throughput-per-node by dividing the backup size by node count. Since this is only now used for that metric, make a failure to count nodes a warning only for release builds (and fallback to 1), and make any error while counting, or not counting to more than 0, a returned error in non-release builds. Release note (bug fix): avoid crashing when BACKUP is unable to count the total nodes in the cluster.
pbardea
approved these changes
Oct 27, 2020
|
bors r+ |
|
Build failed (retrying...): |
|
Build succeeded: |
This was referenced Oct 28, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've seen a report of a node that crashed due to a divide-by-zero
hit during metrics collection, specifically when computing the
throughput-per-node by dividing the backup size by node count.
Since this is only now used for that metric, make a failure to count
nodes a warning only for release builds (and fallback to 1), and make
any error while counting, or not counting to more than 0, a returned
error in non-release builds.
Release note (bug fix): avoid crashing when BACKUP is unable to count the total nodes in the cluster.