Handling of HTTP error response codes in fetcher

[kdhageman]

When using the fetcher, all failed HTTP requests are dumped to glog, without any opportunity for the developer to act on those HTTP messages. For instance, we are running into large amounts of 429 response codes, polluting the logs, and there is no option for us to build some backoff policy ourselves.

The current code base states a TODO for a backoff policy, but it might be useful to let developers can implement their own solution to HTTP response codes for the sake of flexibility.

[pav-kv]

Hello. Could you describe your use-case please to help us find the best solution?

Re custom solutions for HTTP response codes and retry policies. Note that Fetcher accepts a client.LogClient struct. We could make it an interface, which would allow you to implement a decorator with your policies. You would then need to pass this decorated LogClient to the Fetcher, instead of the current undecorated one.

[mhutchinson]

I've submitted #747 to at least make this perform backoff on each attempt. This is healthier for client side log spam, and also protects servers from client abuse.

[pav-kv]

I'm also preparing PR #748 which allows decorating requests. @kdhageman Will it help you to handle retries / errors in a way you want?

[kdhageman]

@pavelkalinnikov I think PR #748 will work fine, thanks!

To elaborate on the use case, in our current specific case we are running into Too Many Request, for which a back-off mechanism would probably be a good solution. There might be a future case though (for instance a CT log returning 5xx response codes), for which a different action is required (e.g. shutting down fetching altogether).

[pav-kv]

You might find Trillian's client/backoff package useful (we use it to implement retry / backoff strategies in this repo, as, for example, in #747).

I'm closing this issue for now. Feel free to reopen if it requires any more action.