Log certain Sidekiq errors only on their final retry #353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pmanrubia
suggested changes
Sep 21, 2017
app/jobs/application_job.rb
Outdated
| class ApplicationJob | ||
| include Sidekiq::Worker | ||
|
|
||
| # List of errors to retry without alerting Sentry until their final retry | ||
| ERRORS_TO_RETRY_WITHOUT_ALERTS = [ | ||
| Google::Apis::RateLimitError, |
There was a problem hiding this comment.
I would not include here Google::Apis::RateLimitError. It could have implications in our GA quota.
| @@ -0,0 +1,3 @@ | |||
| Raven.configure do |config| | |||
There was a problem hiding this comment.
Could you please write a brief description in the commit notes about what raven is, and where we can find documentation about it?
Some of our Sidekiq errors arise from interactions with 3rd party APIs, and failures beyond our control are to be expected from time to time. We do not want to be notified every time we get one of these errors; we just want to be notified the final time the job fails. This change updates our `ApplicationJob` to perform its tasks inside a block that will be rescued when certain errors raise. It will then re-raise a `RetryableError`, which will be ignored by Sentry. When the retries on a job are exhausted, the error will finally be sent to Sentry. Raven is Sentry's API offering. Documentation on the Ruby implementation can be found [here][raven]. We thought about solving this using Raven's `should_capture` method, which would enable us to filter errors base on arbitrary conditions, but that would have require some error parsing. The approach taken here is more explicit about the expected errors, and requires fewer moving parts. [raven]: https://docs.sentry.io/clients/ruby/
alecgibson
force-pushed
the
log-last-sidekiq-retry
branch
from
21e8ebf to
81ea954
Compare
|
@pmanrubia done. |
pmanrubia
approved these changes
Sep 21, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some of our Sidekiq errors arise from interactions with 3rd party APIs,
and failures beyond our control are to be expected from time to time.
We do not want to be notified every time we get one of these errors;
we just want to be notified the final time the job fails.
This change updates our
ApplicationJobto perform its tasks inside ablock that will be rescued when certain errors raise. It will then
re-raise a
RetryableError, which will be ignored by Sentry.When the retries on a job are exhausted, the error will finally be sent
to Sentry.
Raven is Sentry's API offering. Documentation on the Ruby implementation
can be found here.
We thought about solving this using Raven's
should_capturemethod,which would enable us to filter errors base on arbitrary conditions,
but that would have require some error parsing. The approach taken here
is more explicit about the expected errors, and requires fewer moving
parts.