Measure distributor push duration

[zalegrala]

What this PR does:

• Add metric distributor_push_duration

Which issue(s) this PR fixes:
Fixes #614

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Looking good. Added a few requests.

@jpkrohling We are adding this metric as a way to get write latency on Tempo. We cannot measure this more directly b/c we rely on the otelcol receivers.

Do the receivers expose this metric directly in anyway?
Alternatively do the receivers allow for a custom grpc or http server to be used?

[joe-elliott]

unsure on these buckets. 2s is a long time for the smallest one. Our p50 is 50ms on this endpoint in ops. Maybe review what the buckets are in the weaveworks common library since we use those on our other endpoints?

[zalegrala]

I'll have a look. I wasn't quite sure about the bucket counting.

[zalegrala]

Not much in weaveworks, but lots in coretex. After thinking it over and reviewing the defaults, the default seems reasonable Low enough and high enough with reasonable steps in between.

DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}

[jpkrohling]

@jpkrohling We are adding this metric as a way to get write latency on Tempo. We cannot measure this more directly b/c we rely on the otelcol receivers.

I was sure that we had histograms for every component, including receivers. I just ran a test here and it's not there, even with the metrics level set to "detailed". I'll dig for more information, perhaps my recollection is wrong.

For the record: some components have custom metrics and are instrumented using OpenCensus. There's a task going on to replace it with OpenTelemetry Metrics SDK. References:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/dd44390a4a3b64b3a9c9a1ca57e95903282765ac/exporter/loadbalancingexporter/factory.go#L34
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/metrics.go
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/dd44390a4a3b64b3a9c9a1ca57e95903282765ac/exporter/loadbalancingexporter/resolver_dns.go#L125

[joe-elliott]

Oops, one requested change. Can we get an addition to the writes dashboard as well with this metric?

[zalegrala]

Definitely want the dashboard, but I thought I was to do that in the deployment repo. Is the mixin here imported?

[zalegrala]

Alright, I've included an update to the dashboard, but not sure how to test it yet.