Flow: add OpenTelemetry Collector components

[rfratto]

This issue proposes a set of otelcol components to add first-class OpenTelemetry Collector support in Grafana Agent Flow.

Adding first-class OpenTelemetry Collector support enables Grafana Agent to embrace Grafana's big tent philosophy, allowing the agent to support multiple vendors outside of the LGTM stack.

Proposal

We will create a set of OpenTelemetry Collector components across three namespaces:

• otelcol.receiver: A list of OpenTelemetry Collector receivers
• otelcol.processor: A list of OpenTelemetry Collector processors
• otelcol.exporter: A list of OpenTelemetry Collector exporters

A description of how to handle extensions is out of scope for this proposal. However, extensions for at least authentication will be included.

NOTE: Throughout this proposal, I use the term "consumer" in the same sense as OpenTelemetry Collector; a consumer is a component which can consume OpenTelemetry data sent from other components.

Pipeline construction

Grafana Agent Flow creates a pipeline by having each component reference other components where data will be sent. This contrasts OpenTelemetry Collector, where components are defined separately from the pipeline.

For consistency with other Flow components, the set of otelcol components will form a pipeline the same way: by having each component reference other components where data will be sent.

Arguments

All otelcol Flow components which send data to a consumer will support the following arguments:

• output.traces: A list of consumers where traces should be forwarded. Only available if the configured component supports traces.
• output.metrics: A list of consumers where metrics should be forwarded. Only available if the configured component supports metrics.
• output.logs: A list of consumers where logs should be forwarded. Only available if the configured component supports logs.

Exported fields

otelcol Flow components which act as consumers will support the following exported fields:

• input: A consumer for telemetry data. Any telemetry type supported by the OTel component can be sent to this consumer.

Note that there is only one exported field for input for all telemetry types. This simplifies the config, and there doesn't seem to be much value in separating the fields.

Example config

otelcol.receiver.jaeger "default" {
  // Define the server settings where applications 
  // may push Jaeger-formatted traces to. 
  server {
    // Create a gRPC server with the default settings. 
    grpc {} 
  }

  output {  
    traces = [otelcol.processor.batch.default.input] 
  } 
}

otelcol.processor.batch "default" {
  output {
    traces = [otelcol.exporter.logging.default.input]
  }
}

otelcol.exporter.logging "default" {
  log_level = "debug"

  // Exporters cannot forward data to a consumer, 
  // so the "output" field does not exist here. 
}

Flow component list

We will cherry-pick specific OpenTelemetry Collector components from both the otelcol and otelcol-contrib distributions to include in Grafana Agent Flow.

Our initial set of components is minimal, but will grow over time. While we should consider supporting most existing OTel components, we should initially avoid supporting any OTel components whose functionality is covered by an existing Flow component. This will be the case for otelcol-contrib receivers which collide with functionality of prometheus.integration Flow components.

We will start with the following components:

otelcol distribution components:

• otelcol.receiver.otlp: Receives OTLP data over the network.
• otelcol.processor.batch: Batches telemetry data before sending to another consumer.
• otelcol.processor.memory_limiter: Drops telemetry data if memory usage is getting too high.
• otelcol.exporter.otlp: Sends telemetry data via gRPC using OTLP.
• otelcol.exporter.otlphttp: Sends telemetry data via HTTP using OTLP.

otelcol-contrib: distribution components:

• otelcol.receiver.jaeger: Receives telemetry data from Jaeger applications.
• otelcol.receiver.kafka: Receives telemetry data from Kafka.
• otelcol.receiver.opencensus: Receives telemetry data from OpenCensus.
• otelcol.receiver.zipkin: Receives telemetry data from Zipkin.
• otelcol.processor.attributes: Processes span attributes.
• otelcol.processor.spanmetrics: Creates metrics from spans.
• otelcol.processor.tail_sampling: Performs tail sampling against tracing data.
• otelcol.exporter.jaeger: Writes telemetry data to a Jaeger server.

This list of components is based solely on the current exposed functionality of the tracing subsystem within Grafana Agent. As the intent is to embrace the big tent philosophy, we should continue to add more exporters for other vendors over time.

Additionally, we will create Grafana Agent-specific custom components:

• otelcol.exporter.logging: Logs telemetry data in-process.
• otelcol.processor.discovery: Associates incoming spans with a Prometheus discovery target.
• otelcol.processor.servicegraph: Generates a service graph from incoming spans.
• otelcol.receiver.prometheus: Receives Prometheus metrics from a prometheus.scrape Flow component.
• otelcol.exporter.prometheus: Sends Prometheus metrics to a prometheus.remote_write Flow component.

The last two components, otelcol.receiver.prometheus and otelcol.exporter.prometheus, enable interoperability between the Prometheus-specific and OpenTelemetry-specific components, allowing users to freely switch between both as needed.

Implementation

Flow otelcol.* components should work by wrapping around upstream components from OpenTelemetry Collector, using River for configuration. We should not reimplement components that already exist, with the except for the Grafana Agent-specific custom components.

An initial prototype for this was done in #1843. The prototype can serve as a base for the final implementation.

Tasks

This is the list of components to implement for the initial set of components:

• Flow: otelcol.receiver.otlp component #2286
• Flow: otelcol.processor.batch component #2285
• Flow: otelcol.processor.memory_limiter component #2287
• Flow: otelcol.exporter.otlp component #2288
• Flow: otelcol.exporter.otlphttp component #2289
• Flow: otelcol.receiver.jaeger component #2290
• Flow: otelcol.receiver.kafka component #2291
• Flow: otelcol.receiver.opencensus component #2292
• Flow: otelcol.receiver.zipkin component #2293
• Flow: otelcol.processor.attributes component #2294
• Flow: otelcol.processor.spanmetrics component #2295
• Flow: otelcol.processor.tail_sampling component #2296
• Flow: otelcol.exporter.jaeger component #2297
• Flow: otelcol.exporter.logging component #2298
• Flow: otelcol.processor.discovery component #2299
• Flow: otelcol.processor.servicegraph component #2300
• Flow: otelcol.receiver.prometheus component #2301
• Flow: otelcol.exporter.prometheus component #2302
• Flow: otelcol.exporter.loadbalancing component #4068
• Flow: Alternative to Static mode's tracing automatic_logging processor #4261
• Flow: otelcol.processor.spanprocessor component #2612

These components will be gradually introduced across a few releases.

[rfratto]

OpenTelemetry Collector extensions will not be exposed as Flow components.

I'm on the fence about this now. Authentication is handled by extensions, and the way we wrap components doesn't make it easy for otelcol Flow component wrappers to handle their own authentication (see #2343).

There seems to be two choices here:

• Match upstream: Flow components should match upstream behavior and introduce components like otelcol.extension.basic_auth which are then wired up to other components which support authentication.
  • This allows otelcol components to match more closely with upstream.
  • This is easier to implement since it's a direct parallel to the thing we're wrapping.
• Match existing Flow components: Bake in authentication support
  • This allows otelcol components to match more closely with prometheus components.
  • This is more challenging to implement.

[ptodev]

The tracing subsystem in Static mode also supports "load_balancing" and "automatic_logging". Are we going to offer feature parity on those?

The Static mode uses the Collector's loadbalancing exporter directly. Should we add it ot the list of components which we need to port over?

For automatic logging I cannot find an equivalent Collector process so we might have to port our own.

[rfratto]

The tracing subsystem in Static mode also supports "load_balancing" and "automatic_logging". Are we going to offer feature parity on those?

For automatic_logging: otelcol.exporter.logging was supposed to be the equivalent of automatic_logging, even if they don't emit exactly the same format.

For load_balancing: yes, that needs to be added to the list; it was something I missed originally.

[sylvainOL]

Hello @rfratto, I tried otelcol.exporter.logging with normal (just number of traces processed) and detailed (there were \nso I figured out it wasn't good).

So with detailed it should work with Loki (another person in our team is moving from elasticsearch to loki) and Grafana?

thanks for the tool!

[cyrille-leclerc]

@zeitlinger and I faced the same \n as @sylvainOL to verify few OTel logs, it forced me to copy paste the message in a text editor to replace it by real line breaks.
I wish the console output I use for debugging honored line breaks

[ptodev]

Hi @sylvainOL and @cyrille-leclerc, thank you for your feedback. I opened another issue for this: #4261
If you have further comments, please feel free to write them in the dedicated issue.

[sylvainOL]

Hi @ptodev, thanks for the comment, I'll follow it :)

[kago-dk]

@rfratto Any plans to add the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor into Grafana Agent Flow so that we can filter spans from traces https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#filter-spans-from-traces ?

Tracked on #4628

[rfratto]

@kago-dk We're open to it, but this issue is just to track the initial set of OpenTelemetry Collector components which are already available in static mode; for anything net-new we'd want to track it in a separate issue. Would you mind opening a new issue requesting filter processor to be added to flow mode?