component/otelcol/exporter: create exporter component abstraction #2227

rfratto · 2022-09-27T20:14:51Z

This commit introduces a new package, component/otelcol/exporter, which exposes a generic Flow component implementation which can run OpenTelemetry Collector exporters.

There is some stuff left to do for this implementation to be complete:

A Zap logging adapter needs to be created to correctly process logs from OpenTelemetry Collector components.
Component-specific metrics are currently ignored.
Component-specific traces are currently ignored.

All of the above will be done in separate PRs.

As of this commit, there are no registered otelcol.exporter.* components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs.

Related to #2213, and based off of the original prototyping done in #1843.

This commit introduces a new package, component/otelcol/exporter, which exposes a generic Flow component implementation which can run OpenTelemetry Collector exporters. There is some stuff left to do for this implementation to be complete: * A Zap logging adapter needs to be created to correctly process logs from OpenTelemetry Collector components. * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. All of the above will be done in separate PRs. As of this commit, there are no registered `otelcol.exporter.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to grafana#2213.

component/otelcol/exporter/exporter_test.go

rfratto · 2022-09-28T00:21:19Z

component/otelcol/exporter/exporter.go

+			// We may want to put off tracing until we have native tracing
+			// instrumentation from Flow, but metrics should come sooner since we're
+			// already set up for supporting component-specific metrics.


Here's an example of what we can do here once we get around to implementing this:
https://github.com/open-telemetry/opentelemetry-go/blob/example/prometheus/v0.32.1/example/prometheus/main.go#L35

It's going to require updating the OpenTeletry Collector dependency, but we're overdue for that anyway.

component/otelcol/consumer.go

rfratto · 2022-09-28T12:49:20Z

component/otelcol/internal/scheduler/scheduler.go

+// Each OpenTelemetry Collector component has one instance per support
+// telemetry signal, which is why Scheduler supports multiple components. For
+// example, when creating the otlpreceiver component, you would have three
+// total instances: one for logs, one for metrics, and one for traces.
+// Scheduler should only be used to manage the different signals of the same
+// OpenTelemetry Collector component; this means that otlpreceiver and
+// jaegerreceiver should not share the same Scheduler.


@tpaschalis how's this?

Yeah, reads much cleaner, thanks for the explanation!

component/otelcol/internal/scheduler/scheduler.go

component/otelcol/internal/scheduler/host.go

mattdurham

LGTM

tpaschalis · 2022-09-28T14:29:07Z

component/otelcol/consumer.go

+	otelconsumer.Traces
+	otelconsumer.Metrics
+	otelconsumer.Logs
+}


So the way that the OTel collector works here is that an implementation can choose to handle a signal with a consumer, or return ErrDataTypeIsNotSupported, right?

Right, though ErrDataTypeIsNotSupport isn't really intended to be used by consumers; we're just reusing it here for convenience. OpenTelemetry would give you a warning at load time if you're trying to send traces to a component that doesn't support it, but since we're combining everything into one consumer we have emit the errors at runtime instead.

component/otelcol/consumer.go

tpaschalis

LGTM!

This commit introduces a new package, component/otelcol/receiver, which exposes a generic Flow component implementation which can run OpenTelemetry Collector receiver. Like grafana#2227, it leaves some work unfinished for future PRs: * A Zap logging adapter needs to be created to correctly process logs from OpenTelemetry Collector components. * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.receiver.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to grafana#2213.

) This commit introduces a new package, component/otelcol/receiver, which exposes a generic Flow component implementation which can run OpenTelemetry Collector receiver. Like #2227, it leaves some work unfinished for future PRs: * A Zap logging adapter needs to be created to correctly process logs from OpenTelemetry Collector components. * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.receiver.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to #2213.

This commit introduces a new package, component/otelcol/processor, which exposes a generic Flow component implementation which can run OpenTelemetry Collector processor. Like grafana#2227 and grafana#2254, it leaves some work unfinished for future PRs: * A Zap logging adapter needs to be created to correctly process logs from OpenTelemetry Collector components. * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.processor.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to grafana#2213.

…2284) This commit introduces a new package, component/otelcol/processor, which exposes a generic Flow component implementation which can run OpenTelemetry Collector processor. Like #2227 and #2254, it leaves some work unfinished for future PRs: * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.processor.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to #2213.

OpenTelemetry supports many extensions. Extensions are used as a generic way to put "everything else" into OpenTelemetry. There are two types of extensions relevant to us: * [Authentication extensions][auth-ext]: used for both client and server authentication. * [Storage extensions][storage-ext]: used for external storage of state. Other extensions, such as [awsproxy][] are useful but better suited as generic Flow components rather than being shoved in the otelcol namespace, since they are unrelated to telemetry pipelines and aren't referenced by other otelcol components in the upstream configuration. This commit introduces a new package, component/otelcol/auth, which exposes a generic Flow component implementation which can run OpenTelemetry Collector extensions meant for authentication. While storage extensions may end up being Flow components eventually, it's currently marked as experimental upstream. We will reevaluate storage extension components once things have stabilized a little more. Like grafana#2227, grafana#2254, and grafana#2284, it leaves some work unfinished for future PRs: * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.auth.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to grafana#2213. [auth-ext]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/config/configauth [storage-ext]: https://pkg.go.dev/go.opentelemetry.io/collector/extension/experimental/storage [awsproxy]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.61.0/extension/awsproxy

OpenTelemetry supports many extensions. Extensions are used as a generic way to put "everything else" into OpenTelemetry. To quote their [documentation][ext-docs]: > Extension is the interface for objects hosted by the OpenTelemetry > Collector that don't participate directly on data pipelines but provide > some functionality to the service, examples: health check endpoint, > z-pages, etc. There are two types of extensions relevant to us: * [Authentication extensions][auth-ext]: used for both client and server authentication. * [Storage extensions][storage-ext]: used for external storage of state. Other extensions, such as [awsproxy][] are useful but better suited as generic Flow components rather than being shoved in the otelcol namespace, since they are unrelated to telemetry pipelines and aren't referenced by other otelcol components in the upstream configuration. This commit introduces a new package, component/otelcol/auth, which exposes a generic Flow component implementation which can run OpenTelemetry Collector extensions meant for authentication. While storage extensions may end up being Flow components eventually, it's currently marked as experimental upstream. We will reevaluate storage extension components once things have stabilized a little more. Like grafana#2227, grafana#2254, and grafana#2284, it leaves some work unfinished for future PRs: * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.auth.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to grafana#2213. [ext-docs]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/component#Extension [auth-ext]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/config/configauth [storage-ext]: https://pkg.go.dev/go.opentelemetry.io/collector/extension/experimental/storage [awsproxy]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.61.0/extension/awsproxy

…2352) OpenTelemetry supports many extensions. Extensions are used as a generic way to put "everything else" into OpenTelemetry. To quote their [documentation][ext-docs]: > Extension is the interface for objects hosted by the OpenTelemetry > Collector that don't participate directly on data pipelines but provide > some functionality to the service, examples: health check endpoint, > z-pages, etc. There are two types of extensions relevant to us: * [Authentication extensions][auth-ext]: used for both client and server authentication. * [Storage extensions][storage-ext]: used for external storage of state. Other extensions, such as [awsproxy][] are useful but better suited as generic Flow components rather than being shoved in the otelcol namespace, since they are unrelated to telemetry pipelines and aren't referenced by other otelcol components in the upstream configuration. This commit introduces a new package, component/otelcol/auth, which exposes a generic Flow component implementation which can run OpenTelemetry Collector extensions meant for authentication. While storage extensions may end up being Flow components eventually, it's currently marked as experimental upstream. We will reevaluate storage extension components once things have stabilized a little more. Like #2227, #2254, and #2284, it leaves some work unfinished for future PRs: * Component-specific metrics are currently ignored. * Component-specific traces are currently ignored. As of this commit, there are no registered `otelcol.auth.*` components. Implementations for OpenTelemetry Collector Flow components will be done in future PRs. Related to #2213. [ext-docs]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/component#Extension [auth-ext]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/config/configauth [storage-ext]: https://pkg.go.dev/go.opentelemetry.io/collector@v0.61.0/extension/experimental/storage [awsproxy]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.61.0/extension/awsproxy

rfratto commented Sep 27, 2022

View reviewed changes

component/otelcol/exporter/exporter_test.go Outdated Show resolved Hide resolved

rfratto requested review from mattdurham and tpaschalis September 27, 2022 21:27

switch to using JSON reprsentation for test traces

ac00f9e

rfratto commented Sep 28, 2022

View reviewed changes

tpaschalis reviewed Sep 28, 2022

View reviewed changes

component/otelcol/consumer.go Outdated Show resolved Hide resolved

rfratto mentioned this pull request Sep 28, 2022

component/otelcol: set up groundwork for OpenTelemetry Collector components #2224

Merged

rfratto added 2 commits September 28, 2022 08:37

fix typo

9a720ba

scheduler: clarify comment

6ace042

rfratto commented Sep 28, 2022

View reviewed changes

component/otelcol/internal/scheduler/scheduler.go Outdated Show resolved Hide resolved

Update component/otelcol/internal/scheduler/scheduler.go

2cb523e

mattdurham reviewed Sep 28, 2022

View reviewed changes

component/otelcol/internal/scheduler/host.go Outdated Show resolved Hide resolved

scheduler: make customizing exports/extensions options on NewHost

2e061c9

mattdurham approved these changes Sep 28, 2022

View reviewed changes

tpaschalis reviewed Sep 28, 2022

View reviewed changes

component/otelcol/consumer.go Show resolved Hide resolved

tpaschalis approved these changes Sep 28, 2022

View reviewed changes

fix flaky test

04553c2

rfratto merged commit e7ee687 into grafana:main Sep 28, 2022

rfratto deleted the otelcol-exporter branch September 28, 2022 15:17

rfratto mentioned this pull request Sep 30, 2022

component/otelcol/receiver: create receiver component abstraction #2254

Merged

rfratto mentioned this pull request Oct 3, 2022

component/otelcol/processor: create processor component abstraction #2284

Merged

rfratto mentioned this pull request Oct 12, 2022

component/otelcol/auth: create auth extension component abstraction #2352

Merged

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Mar 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Mar 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

component/otelcol/exporter: create exporter component abstraction #2227

component/otelcol/exporter: create exporter component abstraction #2227

rfratto commented Sep 27, 2022

rfratto Sep 28, 2022

rfratto Sep 28, 2022

tpaschalis Sep 28, 2022

mattdurham left a comment

tpaschalis Sep 28, 2022

rfratto Sep 28, 2022

tpaschalis left a comment

component/otelcol/exporter: create exporter component abstraction #2227

component/otelcol/exporter: create exporter component abstraction #2227

Conversation

rfratto commented Sep 27, 2022

rfratto Sep 28, 2022

Choose a reason for hiding this comment

rfratto Sep 28, 2022

Choose a reason for hiding this comment

tpaschalis Sep 28, 2022

Choose a reason for hiding this comment

mattdurham left a comment

Choose a reason for hiding this comment

tpaschalis Sep 28, 2022

Choose a reason for hiding this comment

rfratto Sep 28, 2022

Choose a reason for hiding this comment

tpaschalis left a comment

Choose a reason for hiding this comment