Add stoplight for object storage failures, return HTTP 503

[Gargron]

Reduce traffic congestion due to timeouts

[nightpool]

can we make this opt in through an environment variable? it seems bad if people on reliable storage providers would see increased unavailability due to transient network errors.

i.e., if I'm using Digital Ocean Spaces, and there's a one off timeout, I don't want that to trigger a stoplight, because I know it's probably not indicative of a more serious failure.

[Gargron]

@nightpool Threshold is 3 and cooldown is 60 seconds by default. Can we find a combination that you would be happy with without requiring extra configuration? Even a cooldown of 10 seconds would greatly reduce the amount of timing out concurrent requests.

Edit: Changed threshold to 10 failures, cooldown to 30 seconds. If object storage fails 10 times, the next 30 seconds it will fail instantly, then it can fail another 10 times. Is that OK with you?

[ClearlyClaire]

One thing that this could have enabled is receiving and processing statuses with media attachments when the object storage is down. However, I think the error handling in ActivityPub::Activity::Create#process_attachments would actually not catch it, and abort the whole job?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ClearlyClaire]

Seems fine to me although I'm a bit worried about temporary storage server failures causing to silently avoid downloading media attachments/emoji. It's probably a bit more useful overall, but I'm afraid this would make the behavior slightly more surprising and slightly more difficult to debug.