Retry for subgraph requests

[bnjjj]

Related to #338 and #2006 .

I implements the retry on subgraph requests. Here is an example of configuration:

traffic_shaping:
  all:
    experimental_retry:
      min_per_sec: 10
      ttl: 10s
      retry_percent: 0.2
  subgraphs:
    accounts:
      experimental_retry:
        min_per_sec: 20

Still as an experimental feature because we don't know if it's the right configuration shape. Feel free to give your opinions in this discussion.

[oliverjumpertz]

Some basic feedback after we have used this extensively for quite some time:

Works like a charm.

The only issue is that the configuration is heavily leaned toward the Finneagle algorithm used to implement this feature. It would probably help a lot by clarifying a few of these properties to make it really clear (from reading the config alone, and not the docs) which setting does exactly lead to which behavior. :)

To be honest, though, I would probably have to think for a while myself to come up with better naming, or an idea of how to improve this further.

[bnjjj]

Do you think a more complete documentation or examples would be sufficient ?

[oliverjumpertz]

That would definitely help. :)

I think a few examples would be a pretty good start.

Afair, we asked you for this feature back in the day, when we didn't have a service mesh integrated yet, and partially used internal ingress traffic for TLS termination and such, instead of relying on kube-proxy.

Nginx tends to terminate HTTP 2 connections from time to time (default is after 1000 requests reusing a connection, if I recall correctly), which resulted in a GOAWAY that the router couldn't handle, and thus resulted in an error response.

So, maybe writing a few lines about such use cases and providing examples would already help. :)

[bnjjj]

cc @shorgi we have a lack of docs on retry