Retry with exponential backoff

[enerdgumen]

Does Apalis support retry with exponential backoff?

There is a middleware for Tower that supports this kind of retry (backoff-tower), but I'm wondering if this is the right approach. For example, we can't recover the state of the retry when the server restarts, and the retry execution is not distributed across multiple server instances.

[geofmureithi]

Ideally yes. Apalis supports any Tower specific layer.
There are ways to workaround the specific issues like persistence and distribution.
Apalis provides an Attempt type which stores the number of attempts. You can pair this with a layer that calls Storage::update_with_id when the request is fails. This layer would need to be called before the backoff layer, and you might need to look at the current Policy for retry layer for some pointers on this.
In short, its technically possible, but some extra work is needed.
I am willing to help you achieve this, you could just give it a try.

[enerdgumen]

Thinking about this... The point is that Tower is designed to support request/response patterns, and long running retries don't seem appropriate in this case. For example, if at a certain point you need to retry some days later, does it make sense to keep the pending task in memory for all the time?
Is maybe more appropriate to manually schedule a new job in case of failure?

[geofmureithi]

This is correct. Apalis storage offers a reschedule method.

You could port the backoff crate and add specific logic where you update the attempts (on fail) then use the method above.
It is entirely achievable but needs some specific work