docs: adds instructions for collecting and sending native histograms with otel collector #9328
Conversation
…with otel collector
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
|
|
||
| For more information, refer to [Registering Views](https://opentelemetry.io/docs/languages/go/instrumentation/#registering-views) in the OpenTelemetry SDK documentation for Go. | ||
|
|
||
| ## Migrate from classic histograms |
There was a problem hiding this comment.
@krajorama do the migration instructions at https://grafana.com/docs/mimir/next/send/native-histograms/#migrate-from-classic-histograms not apply here? If that's the case, how can people migrate?
There was a problem hiding this comment.
The migration starts differently, but from the dashboards (step 5) is the same.
In OTEL the explicit bucket and exponential bucket histograms cannot easily co-exist as the name of the metric would clash. So you start from an existing histogram that uses explicit buckets and add a new view where you assign a new name as well as exponential aggregation to the histogram. This will result in a new metric with the new name and exponential buckets. For example I have in my test code a histogram called request_latency and I can add a new view that will create request_latency_exp metric with exponential buckets.
v := sdkmetric.NewView(sdkmetric.Instrument{
Name: "request_latency",
Kind: sdkmetric.InstrumentKindHistogram,
}, sdkmetric.Stream{
Name: "request_latency_exp",
Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20},
//Aggregation: sdkmetric.DefaultAggregationSelector(sdkmetric.InstrumentKindHistogram),
})
|
|
||
| ## Migrate from classic histograms | ||
|
|
||
| ## Bucket boundary calculation |
There was a problem hiding this comment.
@krajorama same question as above.
There was a problem hiding this comment.
Almost the same, except the bucket offsets are shifted by one, so in native histogram you have
but in otel you have
This isn't terribly interesting for end-users :) See also https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/4a5e6db494acb2d56617266828f8db03957d3960/pkg/translator/prometheusremotewrite/histograms.go#L107
| }, sdkmetric.Stream{ | ||
| Name: "request_latency_exp", | ||
| Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, | ||
| //Aggregation: sdkmetric.DefaultAggregationSelector(sdkmetric.InstrumentKindHistogram), |
There was a problem hiding this comment.
woops, copy paste from my test code, this is a comment, can be removed
| //Aggregation: sdkmetric.DefaultAggregationSelector(sdkmetric.InstrumentKindHistogram), |
|
|
||
| 1. After configuring exponential histogram collection, choose one of the following ways to stop collecting explicit bucket histograms. | ||
|
|
||
| - Remove the custom bucket definition, `Buckets`/`classicUpperBounds`, from the instrumentation. In Java, also use the `nativeOnly()` option. Refer to the examples in [Instrument application with Prometheus client libraries](#instrument-application-with-prometheus-client-libraries). |
There was a problem hiding this comment.
This "Remove the custom bucket definition" is Prometheus specific, I would not go into details, just mention to remove explicit bucket definition and/or view that exposes the explicit buckets.
| 1. After configuring exponential histogram collection, choose one of the following ways to stop collecting explicit bucket histograms. | ||
|
|
||
| - Remove the custom bucket definition, `Buckets`/`classicUpperBounds`, from the instrumentation. In Java, also use the `nativeOnly()` option. Refer to the examples in [Instrument application with Prometheus client libraries](#instrument-application-with-prometheus-client-libraries). | ||
| - Drop the explicit bucket histogram series with [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or [Grafana Alloy prometheus.relabel](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/prometheus/prometheus.relabel) at the time of scraping. |
There was a problem hiding this comment.
I haven't tried it but in the Otel collector you have something like filter processor that should be possible to utilize for the same purpose. Let's not mix Prometheus or Alloy here for now.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
|
|
||
| For more information about creating a view, refer to [View](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. | ||
|
|
||
| 1. Modify dashboards to use the exponential histogram metrics. Refer to [Visualize native histograms](https://grafana.com/docs/mimir/<MIMIR_VERSION>/visualize/native-histograms/) for more information. |
There was a problem hiding this comment.
nit: maybe make common text with the other doc reusable/common in a followup PR if possible so the two docs don't deviate?
| The availability of different receivers and exporters depends on your OpenTelemetry Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). | ||
| {{< /admonition >}} | ||
|
|
||
| You can use the OpenTelemetry protocol (OTLP) to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir/<MIMIR_VERSION>/send/native-histograms/). |
There was a problem hiding this comment.
We did not document sending with OTEL collector, so let's remove the reference. Also I realized when looking into this that there's two OTLP exporters and we only support the one over HTTP protocol.
| You can use the OpenTelemetry protocol (OTLP) to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir/<MIMIR_VERSION>/send/native-histograms/). | |
| You can use the OpenTelemetry protocol (OTLP) over HTTP to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. |
We could refer to the the OTLP HTTP exporter and Promtheus remote write exporter documentation:
What this PR does
This PR updates https://grafana.com/docs/mimir/next/send/native-histograms/ with instructions for collecting and sending native histograms with the OTel Collector.
Which issue(s) this PR fixes or relates to
Fixes https://github.com/grafana/mimir-squad/issues/2131)](https://github.com/grafana/mimir-squad/issues/2131
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.