Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block builder: update alerts and add critical alerts #10270

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

codesome
Copy link
Member

What this PR does

Updated some warnings and add critical alerts to block builders.

BlockBuilderNoCycleProcessing is a little sensitive. Meaning if a consumption ran a little longer than 5m compared to the earlier one (for duration is 5m), this alert will fire. This sensitivity is fine for warnings, but we can up it to 10 mins to reduce any potential noise. And for critical alert, we can probably a for duration of 20 mins.

BlockBuilderLagging current warning alert fires when the lag is over 4M for 75mins (basically lag went above 4M, but it stayed above 4M in the next cycle as well). We have a critical alert with for duration of 2h20m. Meaning once it went above 4M in a cycle, it remained about 4M in the next two cycles as well. If the cycle run was not successful, then BlockBuilderNoCycleProcessing will already fire much the lagging alert will fire. BlockBuilderLagging will catch other issues that is not just "not able to consume", like block builders are not scaled enough for example.

If BlockBuilderCompactAndUploadFailed fired, it should probably be a critical alert immediately.

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant