feat: Allow total duration with max retries

[tl-marco-tormento]

This PR fixes one main problem, which is the fact that using a total duration we are not able to set a limit to the number of retries that will be made inside that timeframe.

We added a new build_with_total_retry_duration_and_max_retries method to ExponentialBackoffBuilder that calculates the number of retries that should be made if jitter 1.0 was applied (which means no jitter).
Since jitter can be 0..1, the calculated number of retries is effectively the max number of retries that can be made during that duration.

In should_retry we now check if either total duration or max attempts are exceeded.

We also made the exponential's base configurable: this will allow users to exponentially delay the retries more from one another.
The default remains 2, so this isn't a breaking change.

[ThomWright]

I'm wondering if there's a more general-purpose API design here.

I'm thinking about:

1. What if we want to set these independently?
2. What if we want to calculate duration from max retries? Is that a sensible option?

For 1, what do you think about an API like:

let exponential_backoff_timed = ExponentialBackoff::builder()
    .retry_bounds(Duration::from_secs(1), Duration::from_secs(6 * 60 * 60))
    .max_retries(10)
    .build_with_total_retry_duration(Duration::from_secs(24 * 60 * 60));

or:

ExponentialBackoff::builder()
    .build_with_max_retries(5)
    .with_total_retry_duration(Duration::from_secs(24 * 60 * 60));

Not sure about 2 yet, but something like this:

I might be overcomplicating it, but keen to try to find a good general-purpose API which can cover these cases, instead of committing to specific-purpose APIs.

[tl-marco-tormento]

I'm thinking about:

1. What if we want to set these independently?

What if we provide this API:

fn build_with_total_retry_duration(
        self,
        total_duration: Duration,
        max_retries: MaxRetries
    ) -> ExponentialBackoffTimed;

enum MaxRetries {
       Unbounded,
       Default,
       Fixed(u16)
}

User could:

• select total_duration and MaxRetries::Unbounded => do any number of retries inside total_duration
• select total_duration and MaxRetries::Default => we calculate max number of retries using total_duration and enforce whatever comes first
• select total_duration and MaxRetries::Fixed(max_retries) => we enforce whatever comes first, max_retries is capped at the number we calculate using total_duration anyway

2. What if we want to calculate duration from max retries? Is that a sensible option?

I think that should be the default when using the build_with_max_retries API, because I can't think of a case where I would want to set max_retries without capping the total_duration... do you have any example where that would be sensible?

[ThomWright]

Is this a good illustration of how it will work?

What if we provide this API:

I like it. I think we might want to either find a better name for Default, or describe it really well in the doc comments.

A downside is that it's a breaking change?

I can't think of a case where I would want to set max_retries without capping the total_duration... do you have any example where that would be sensible?

If you constrain both the min and max intervals, you get a fairly good idea of how long something could take... but only as long as the task is short, which is maybe a key oversight!

I'm coming around to this idea.

[tl-marco-tormento]

Is this a good illustration of how it will work?

Yes, that's how the MaxRetries::Fixed(max_retries) would work.

I think we might want to either find a better name for Default

Big issues with naming here...

A downside is that it's a breaking change?

It is, we would deprecate the current build_with_max_retries and build_with_total_duration methods and provide the new API.

If you are onboard I'll start the implementation first thing tomorrow morning.