Add retry to some abuse tasks

[eviljeff]

Fixes: mozilla/addons#15359

Description

Adds retry to some tasks (as specified in the issue). Currently the settings are:

• retry_jitter: True (by default) - all retry delays are variable, with the exact delay between 0 seconds and whatever the retry delay was calculated as.
• retry_backoff: 30 - the first retry will be (a maximum of) 30 seconds after; the retry after that (a maximum of) 60 seconds, then 120, and so on.
• retry_backoff_max: 60*60 - the maximum delay between retries will be 1 hour
• max_retries: 79 - I calculated this with a spreadsheet, to see how many retries would be needed to retry for 72 hours. Though with retry_jitter it will be always less than 72 hours... so could need increasing.

A report of an entity with many relationships can lead to multiple requests - the patch currently excludes those failures from retry as there's no easy way to retry part of a task.

Testing

• Prevent cinder from being accessible from your local addons-server instance somehow (turn off your wifi?)
• Take an action that would normally communicate with Cinder, e.g. submitting an abuse report
• see in logs from addons-server-worker container that the task failed with a request exception
• wait 30 seconds
• see another request exception from celery
• reconnect your connection to Cinder
• after some amount of time (the 1st retry would be up to 30 seconds after the original attempt; the second up to 60 seconds after the first) the task should succeed
• verify the abuse report was submitted and we updated the AbuseReport with an fk to a CinderJob (there may not have been a new CinderJob created if that add-on was reported for abuse previously).

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[eviljeff]

Whether max_retries should be directly defined in the task kw, or inside retry_kwargs was ambiguous - and I'm not sure we do it consistently in our code either. I went with this way because of the 2nd example in https://docs.celeryq.dev/en/latest/userguide/tasks.html#automatic-retry-for-known-exceptions

[diox]

AFAICT from the logs, the problem with autoretry is that the exceptions before we give retrying are not logged as errors. They are just logged as info. (the severity - lowercase - is 200 on each attempt that is retried, 500 on the one where we give up)

That would render those exceptions completely invisible to us for 72 hours, they wouldn't be sent to Sentry.