-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sentry #1461
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dconnolly
added
A-infrastructure
Area: Infrastructure changes
and removed
S-blocked
Status: Blocked on other tasks
labels
Dec 9, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Partly to fullfill #1286, but in general, long-running sync tests don't fit well into testing frameworks because 1: they run very long, and 2: if the objective is 'sync to tip without panicking/erroring', that is basically the main use case. Therefore instead of running a 'test case', we should run zebrad, appropriately configured, and alert on panic/error/crash as a failure, and be quiet if nothing goes wrong (ie, there is never a endpoint for a 'test' like this to pass, only a possible 'fail').
Solution
When enabled, catches panics and other errors and reports them to our Sentry.io project. From there we can set up alerting, webhooks, and automatic GitHub issue opening if we want. Sentry support is behind a feature flag so by default zebrad doesn't include it, and when enabled, pulls the config information about where to send events from the runtime environmental variable. It's enabled for our Docker.build deployments which is used to deploy automatically from the #main branch on every merge and on any Manual Deploy workflow runs.
Review
Related Issues
#1286
Follow Up Work
Twiddle the alerting/webhook/GitHub integration configs to our liking but that does not require any code changes.
Inject our dsn more smartly via github secrets management through the workflow into github container execution environment, but this (inline in the thin release image) works fine for now, can be rotated later.
I can't tell if this works on macOS or not, I've only successfully triggered events/caught panics on Debian with ca-certificates package installed at runtime.