[exporter/loadbalancing] couldn't find the exporter for the endpoint bug

[Frapschen]

Component(s)

exporter/loadbalancing

What happened?

Description

Using this exporter encounters a bug. I have tried dsn and k8s resolver; both cause bugs.

use dns resolver, config:

  loadbalancing/jaeger:
    routing_key: "traceID"
    protocol:
      otlp:
        # all options from the OTLP exporter are supported, except the endpoint
        timeout: 1s
        tls:
          insecure: true
        sending_queue:
          enabled: true
          storage: file_storage/all_settings
        retry_on_failure:
          enabled: true
          max_elapsed_time: 500s
    resolver:
      dns:
        hostname: jaeger-collector-headless.insight-system.svc.cluster.local
        port: "4317"
        interval: 1s
        timeout: 200ms

It's error logs:

using k8s resolver config:

  loadbalancing/jaeger:
    routing_key: "traceID"
    protocol:
      otlp:
        # all options from the OTLP exporter are supported, except the endpoint
        timeout: 1s
        tls:
          insecure: true
        sending_queue:
          enabled: true
          storage: file_storage/all_settings
        retry_on_failure:
          enabled: true
          max_elapsed_time: 500s
    resolver:
      k8s:
        service: insight-jaeger-collector
        ports:
          - 4317

It's error logs:

I have noticed this:

opentelemetry-collector-contrib/exporter/loadbalancingexporter/loadbalancer.go

Line 236 in 21208c0

// something is really wrong... how come we couldn't find the exporter??

It also shows that something is wrong with the code.

Collector version

v0.109.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Please provide your complete configuration file and indicate the version of the collector used.

[jpkrohling]

It looks like I can reproduce this with a fairly simple config, like:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol-loadbalancer
spec:
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.114.0
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}

    exporters:
      loadbalancing:
        protocol:
          otlp:
            timeout: 1s
            tls:
              insecure: true
            sending_queue:
              enabled: true
              storage: file_storage/all_settings
            retry_on_failure:
              enabled: true
              max_elapsed_time: 500s
        resolver:
          dns:
            hostname: otelcol-backend-collector-headless

    service:
      extensions: [ ]
      pipelines:
        traces:
          receivers:  [ otlp ]
          processors: [  ]
          exporters:  [ loadbalancing ]

And with this backend:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol-backend
spec:
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.114.0
  replicas: 10
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}

    exporters:
      debug: {}

    service:
      extensions: [ ]
      pipelines:
        traces:
          receivers:  [ otlp  ]
          processors: [  ]
          exporters:  [ debug ]

This seems to be connected to the persistent sending queue, as removing it confirms that it works.
When everything is working, the following metrics are available:

# HELP otelcol_loadbalancer_backend_outcome Number of successes and failures for each endpoint.
# TYPE otelcol_loadbalancer_backend_outcome counter
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.31:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.32:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.33:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.34:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.35:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.36:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.37:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.38:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.39:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261
otelcol_loadbalancer_backend_outcome{endpoint="10.42.0.40:4317",service_instance_id="19534c16-77db-4e3d-892a-82b65ed1db09",service_name="otelcol-contrib",service_version="0.114.0",success="true"} 2261

With the sending queue, all spans end up being refused, like this:

otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="e3fc5786-a50e-4df2-bd6f-71167e37a26e",service_name="otelcol-contrib",service_version="0.114.0",transport="grpc"} 1536

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.