Block builder: update alerts and add critical alerts #10270
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Updated some warnings and add critical alerts to block builders.
BlockBuilderNoCycleProcessing
is a little sensitive. Meaning if a consumption ran a little longer than 5m compared to the earlier one (for duration is 5m), this alert will fire. This sensitivity is fine for warnings, but we can up it to 10 mins to reduce any potential noise. And for critical alert, we can probably a for duration of 20 mins.BlockBuilderLagging
current warning alert fires when the lag is over 4M for 75mins (basically lag went above 4M, but it stayed above 4M in the next cycle as well). We have a critical alert with for duration of 2h20m. Meaning once it went above 4M in a cycle, it remained about 4M in the next two cycles as well. If the cycle run was not successful, thenBlockBuilderNoCycleProcessing
will already fire much the lagging alert will fire.BlockBuilderLagging
will catch other issues that is not just "not able to consume", like block builders are not scaled enough for example.If
BlockBuilderCompactAndUploadFailed
fired, it should probably be a critical alert immediately.Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]
.about-versioning.md
updated with experimental features.