Rework the connection management

[ansiwen]

There are a lot of issues regarding connection and retry management. So let's discuss here what would be a good scheme.

Requirements:

1. works well with single instance
2. works well with multiple instances
3. multiple instances should be load-balanced
4. failing instances should not slow down, if other instances are still working
5. all instances should have been at least once tried, before giving up a request
6. if all instances are failing, it should not take forever to fail (global timeout?)

some ideas to solve 4:

• a request attempt to one instance should fail quite aggressively, after a request_timeout of maybe only 1-2 seconds. (configurable?)
• if request attempt to an instance fails, all connections to that instance must be killed (firewalls)
• the same instance should not be retried before all other instances have been tried
• for following requests a failing instance should be removed from the active instance pool until a penalty_timeout (with exponential rise until a reasonable maximum)
• retries (before the penalty_timeout is reached) are only applied when all instances are in failing state, that's when the retries config comes into account. that's also when less aggressive retry_timeout comes into account, because we are anyway in a disaster recovery.

so basically we would have two global states:

• working: at least one instance does not have a penalty timeout -> fast operation (only occasional retries after penalty_timeout is reached, but that is failing fast with the aggressive request_timeout)
• failing: all instances have a penalty timeout -> retry mechanism is used, which tries retries * instances times, maximum waiting time per request is retries * instances * retry_timeout.

Related issues: #178 #201 #202 #203 #204 #206 #207 #209 #210 #211

[sosthene-nitrokey]

There are some limitations to what we can do here, mainly limited by the use of synchronous APIs and how ureq works:

if request attempt to an instance fails, all connections to that instance must be killed (firewalls)

There is not an easy way to do that besides dropping and recreating the Ureq agent.

the same instance should not be retried before all other instances have been tried

Ureq also performs a non-blocking read to pooled connection to check that they have not been killed. If this fails it tries a new connection immediately without an override. I think this behaviour is desired however to detect the case that the connection has been (properly) closed by the other side.

for following requests a failing instance should be removed from the active instance pool until a penalty_timeout (with exponential rise until a reasonable maximum)

This attempt needs to be synchronous, so it will slow down some requests. We might be able to do this retry in a background thread in some cases.

The issue is that this would not solve the case of the firewall closing connections and slowing down the requests after that because the first request has failed.

We also need to proactively detect failed connections:

• Enable TCP keepalive -> this helps prevent the firewall from dropping the connection, and detecting when it does so
• Automatically drop connections that have been idle for a while. This cannot be done with Ureq current. Our fork and the 3.x.x branch will support being able to discard old connections instead of trying them when performing a request, but it still means that if the PKCS11 module is untouched, the connections will be considered opened by the kernel.

Otherwise I agree with this approach, but it would need some pretty large refactoring to implement.