Create new tools for http.Handler instrumentation

[beorn7]

As mentioned in the doc comment, InstrumentHandler has issues beyond repair.
In the promhttp package, new utilities should be provided to instrument handlers and round-trippers in a more specific way.

See #200, #101, #57 for other issues to solve on the way.

[beorn7]

https://blog.golang.org/http-tracing and https://github.com/davecheney/httpstat might be useful to look at.

[orian]

Hi. I need this, is anyone else working on it? If not, I'll try to work on it.

[beorn7]

@stuartnelson3 is, kind of. And it already became apparent how hard this is to get right and how much context is needed. Not sure if it makes any sense at this time to involve somebody who misses that context.
If you are creating something like this anyway, no harm in sharing, and if it fits this library, we'll happily integrate.

[orian]

@beorn7 Sure, I understand.

@stuartnelson3 Is this context to read somewhere? Is the current work available somewhere?

Edit: I see you both work at SoundCloud so I guess it was chatted about?

[stuartnelson3]

That is correct, context was shared with me face-to-face. I'll try to look at this soon, but as is recommended in the library, rolling your own custom handler, at this point, will probably be a much quicker solution.

[orian]

Just for others (following comment in: https://github.com/prometheus/client_golang/blob/master/prometheus/http.go#L152):

• switching to seconds is easy;
• switching to histogram is easy, the problem may be how to choose bucket sizes. I've decided to use {0.1, 0.2, ..., 1.0, 2.0, ... 10.0}; I guess most people expect their service to serve response strongly under 1 sec;
• keeping the Summary for a given instance may make sense;
• calculating the size of Request is not trivial as the Handler/HandleFunc gets parsed Request, the best would be to modify the go http.Server to accumulate the size (it's returned anyway during read phase);

The links provided by by @beorn7 are about the client tracing and required to modify the net/http.Transport

You can find my instrumented handler at: https://github.com/orian/go-http-instrument