Allow Exporting of the Collector's internal spans #6629

TylerHelmuth · 2022-11-28T21:31:01Z

Is your feature request related to a problem? Please describe.
Currently the collector can be made to emit metric and log telemetry, but not spans. The collector does have the capability to generate its own spans, but they can only be viewed with the zpages extension. It would be best if the spans could also be emitted like the collector's metrics and logs

Describe the solution you'd like
The collector's telemetry settings allow for configuring a "output" for spans that result in the spans being sent through a collector pipeline.

I think it would be best if the data could be sent to a pipeline directly without needing to go over the network. If that isn't possible then maybe we restrict exporting of spans to the OTLP format and require a OTLP receiver to collect the collector's spans?

The configuration could look something like:

// TracesConfig exposes the common Telemetry configuration for collector's internal spans.
// Experimental: *NOTE* this structure is subject to change or removal in the future.
type TracesConfig struct {
	// Propagators is a list of TextMapPropagators from the supported propagators list. Currently,
	// tracecontext and  b3 are supported. By default, the value is set to empty list and
	// context propagation is disabled.
	Propagators []string `mapstructure:"propagators"`

	// Outputs is a list of pipeline names that should receive the collector's spans.
	// If no outputs are specified then spans are not exported from the collector.
	Outputs []string `mapstructure:"outputs"`
}

service:
  telemetry:
    traces:
      outputs:
        - "pipeline"

Maybe we could use the default pipeline if no outputs are specific, essentially making span exporting the default behavior, and include some other config option to disable span exporting?

jpkrohling · 2022-11-29T14:19:27Z

If the OTLP receiver and exporter are instrumented with tracing and exporting is the default behavior, wouldn't it just cause an endless stream of spans by default?

TylerHelmuth · 2022-11-29T15:27:25Z

@jpkrohling maybe the solution is being able to define in the telemetry settings a pipeline specific to the collector's internal telemetry and this pipeline doesn't instrument itself (somehow)? Could look like:

service:
  telemetry:
    pipeline:
      traces:
        processors: []
        exporters: []
      metrics:
        processors: []
        exporters: []
      logs:
        processors: []
        exporters: []

jpkrohling · 2022-11-29T16:46:22Z

I have a bad feeling about this one. The common practice I've been seeing is to have the monitors being external to the process being monitored, and I think this applies here as well ("who monitors the monitor" type of question). In that sense, I would recommend our users use an external OTLP endpoint whenever possible to receive the collector's telemetry data and would not have this enabled by default.

TylerHelmuth · 2022-11-29T17:00:14Z

@jpkrohling an external endpoint is ok as well, although we lose potential processing power. As long as we can get the spans out I'll be happy.

TylerHelmuth · 2022-11-30T17:20:05Z

Based on the SIG discussion today I will move forward with a config proposal to allow configuring exporters in the OpenTelemetry Go SDK.

TylerHelmuth · 2022-11-30T19:08:03Z

@bogdandrutu @jpkrohling @codeboten @Aneurysm9 was the idea of using ENV VARs to configure the exporter(s) completely rejected? If they are allowed, which takes precedence, env vars or the explicit configuration in TelemetrySettings?

bogdandrutu · 2022-12-01T05:28:55Z

I think we should not support that for the moment, idea being that these kind of decisions should be made by the config SIG and by not supporting I believe that we are postponing that decision.

TylerHelmuth · 2022-12-01T17:34:54Z

Current Requirements:

The configuration must be generic enough to apply to any type of exporter (even if we only support OTLP at the start)
Should be able to be extended to include metrics and logs in the future.
ENV VARs should not be used to do the configuring (for now).

SDK Exporter Configuration

The Go SDK exporters all use the Go pattern of Options to configure the exporter.
These Options vary by exporter type.
Each option can have different parameters

To be able to configure the SDK we will need to be able to identify which options should be used and the parameters

Collector TelemetrySettings Configuration Proposal

I think we can handle mapping from configuration to Option via a combination of the exporter’s name and the option name.

service:
  telemetry:
    traces:
      exporters:
        - name: "exporter name"
          options: 
            - name: "option name"
              parameters: 
                - "string"
                - "string"
            - name: "option name"
              parameters: 
                - "string"
                - "string"
        - name: "exporter name"
          options: 
            - name: "option name"
              parameters: 
                - "string"
                - "string"
            - name: "option name"
              parameters: 
                - "string"
                - "string"

Downsides:

User must know the Go SDK in order to configure the collector’s use of the Go SDK. If the user doesn’t know what exporters/options are available (and no using Intelisense here to help) it will be difficult to configure.
Maybe doesn’t extend to other languages, won’t fit in with the Configuration SIG’s goals?

Upsides:

Generic enough to handle any type of exporter, as long as it uses options.
Explicit; all options are available and configurable.
Scalable; as more options/exporters are added the configuration doesn’t have to change

jpkrohling · 2022-12-05T12:05:14Z

Are we allowing users to configure other settings, like samplers, tracer names, and custom resource attributes? I think it would also be good to have a property to enable tracing, like:

service:
  telemetry:
    traces:
      enabled: true

TylerHelmuth · 2022-12-05T15:42:54Z

@jpkrohling we don't currently have an enable switch for metrics or logs or the traces that are created today. If we add one for tracing then I think it should be consistent and we should add some for metric and log telemetry as well. The lack of any exporter configuration could also be used to signify spans should not be exported but should continue to be created and exposed via zpages.

I think we will need to be able to configure samplers and tracer name (although I think we should supply defaults for both). Custom resource attributes should reuse the telemetry setting's existing Resource field I think.

codeboten · 2023-01-03T23:12:47Z

@TylerHelmuth adding the current example configuration from the configuration working group since we're essentially configuring the OTel Go SDK, I would lean towards supporting the same configuration:

  tracer_provider:
    # Span exporters. Each exporter key refers to the type of the exporter. Values configure the exporter. Exporters must be associated with a span processor.
    exporters:
      # Configure the zipkin exporter.
      zipkin:
        # Sets the endpoint.
        #
        # Environment variable: OTEL_EXPORTER_ZIPKIN_ENDPOINT
        endpoint: http://localhost:9411/api/v2/spans
        # Sets the max time to wait for each export.
        #
        # Environment variable: OTEL_EXPORTER_ZIPKIN_TIMEOUT
        timeout: 10000
    # List of span processors. Each span processor has a name and args used to configure it.
    span_processors:
      # Add a batch span processor.
      #
      # Environment variable: OTEL_BSP_*, OTEL_TRACES_EXPORTER
      - name: batch
        # Configure the batch span processor.
        args:
          # Sets the delay interval between two consecutive exports.
          #
          # Environment variable: OTEL_BSP_SCHEDULE_DELAY
          schedule_delay: 5000
          # Sets the maximum allowed time to export data.
          #
          # Environment variable: OTEL_BSP_EXPORT_TIMEOUT
          export_timeout: 30000
          # Sets the maximum queue size.
          #
          # Environment variable: OTEL_BSP_MAX_QUEUE_SIZE
          max_queue_size: 2048
          # Sets the maximum batch size.
          #
          # Environment variable: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
          max_export_batch_size: 512
          # Sets the exporter. Exporter must refer to a key in sdk.tracer_provider.exporters.
          #
          # Environment variable: OTEL_TRACES_EXPORTER
          exporter: zipkin

See https://github.com/MrAlias/otel-schema/blob/332987fc1980b2631122771c8d721fa3e7ab32d4/config.yaml#L35-L77 for additional configuration

TylerHelmuth · 2023-01-05T15:49:18Z

@codeboten I'm all for using what the Working Group is creating. 1 question, is the trace_provider.exporters section a list or a map? Would the collector's implementation need to update any time we need to support a new exporter?

codeboten · 2023-01-05T16:26:47Z

@codeboten I'm all for using what the Working Group is creating.

💯, otherwise i imagine this would be confusing for users. Although tbh, there are a lot of ways this exporter configuration could be confusing to users. I'm already asking myself the question why is this configuration different than the other exporters 🥲

1 question, is the trace_provider.exporters section a list or a map?

The exporters section is currently a map. I think there was talk of supporting a similar pattern as the collector's zipkin/somewhere zipkin/somewhereelse to identify multiple instances of the same exporter with different configuration.

Would the collector's implementation need to update any time we need to support a new exporter?

I would expect this to be handled by the otel go sdk, so if the otel go sdk supports a new exporter, then the collector would need to update the dependency.

TylerHelmuth · 2023-01-09T23:02:02Z

@codeboten with that configuration which fields does the working group think should be required? Does the working group expect certain fields to have default values?

Default values may be tricky for us. At the moment we are using the presence of tracer_provider to mean that the traces should be exported. I think it will be hard to detect when the user has specified tracer_provider but not all the fields and then default the un-supplied fields.

As an inital go at this feature it may be easiest to require the user to explicitly specify all config fields they need.

atoulme · 2023-12-19T04:53:37Z

It looks to me that open-telemetry/opentelemetry-collector-contrib#28994 indicates this issue is now resolved. Anything left to do here?

…o beta (#11091) #### Description This configuration of the internal collector telemetry has been using the SDK configuration for some time. Moving this gate from alpha to beta. #### Link to tracking issue Related to #6629 --------- Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>

…o beta (open-telemetry#11091) #### Description This configuration of the internal collector telemetry has been using the SDK configuration for some time. Moving this gate from alpha to beta. #### Link to tracking issue Related to open-telemetry#6629 --------- Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>

TylerHelmuth changed the title ~~Allow Exporting of the Collector's internal tracing~~ Allow Exporting of the Collector's internal spans Nov 28, 2022

jpkrohling mentioned this issue Nov 30, 2022

Consider ability to specify a noop, heavily sampled or restricted zap logger for a pipeline #6635

Closed

TylerHelmuth mentioned this issue Feb 21, 2023

Proposed strategy for making the collector an observable application #7223

Closed

codeboten mentioned this issue Apr 12, 2023

Proposed strategy for making the collector an observable application #7532

Open

5 tasks

sirianni mentioned this issue Jun 28, 2024

Provide mechanism to disable internal tracing #10481

Open

codeboten mentioned this issue Sep 9, 2024

[service] move useOtelWithSDKConfigurationForInternalTelemetry gate to beta #11091

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow Exporting of the Collector's internal spans #6629

Allow Exporting of the Collector's internal spans #6629

TylerHelmuth commented Nov 28, 2022 •

edited

Loading

jpkrohling commented Nov 29, 2022

TylerHelmuth commented Nov 29, 2022

jpkrohling commented Nov 29, 2022

TylerHelmuth commented Nov 29, 2022 •

edited

Loading

TylerHelmuth commented Nov 30, 2022

TylerHelmuth commented Nov 30, 2022 •

edited

Loading

bogdandrutu commented Dec 1, 2022

TylerHelmuth commented Dec 1, 2022 •

edited

Loading

jpkrohling commented Dec 5, 2022

TylerHelmuth commented Dec 5, 2022 •

edited

Loading

codeboten commented Jan 3, 2023

TylerHelmuth commented Jan 5, 2023

codeboten commented Jan 5, 2023

TylerHelmuth commented Jan 9, 2023

atoulme commented Dec 19, 2023

Allow Exporting of the Collector's internal spans #6629

Allow Exporting of the Collector's internal spans #6629

Comments

TylerHelmuth commented Nov 28, 2022 • edited Loading

jpkrohling commented Nov 29, 2022

TylerHelmuth commented Nov 29, 2022

jpkrohling commented Nov 29, 2022

TylerHelmuth commented Nov 29, 2022 • edited Loading

TylerHelmuth commented Nov 30, 2022

TylerHelmuth commented Nov 30, 2022 • edited Loading

bogdandrutu commented Dec 1, 2022

TylerHelmuth commented Dec 1, 2022 • edited Loading

Current Requirements:

SDK Exporter Configuration

Collector TelemetrySettings Configuration Proposal

Downsides:

Upsides:

jpkrohling commented Dec 5, 2022

TylerHelmuth commented Dec 5, 2022 • edited Loading

codeboten commented Jan 3, 2023

TylerHelmuth commented Jan 5, 2023

codeboten commented Jan 5, 2023

TylerHelmuth commented Jan 9, 2023

atoulme commented Dec 19, 2023

TylerHelmuth commented Nov 28, 2022 •

edited

Loading

TylerHelmuth commented Nov 29, 2022 •

edited

Loading

TylerHelmuth commented Nov 30, 2022 •

edited

Loading

TylerHelmuth commented Dec 1, 2022 •

edited

Loading

TylerHelmuth commented Dec 5, 2022 •

edited

Loading