prometheusexporter's "metric_expiration" parameter is ignored when used with spanmetrics connector

[tqi-raurora]

Component(s)

exporter/prometheus

What happened?

Description

metric_expiration setting for prometheus exporter seems to be ignored

Using otelcol-contrib version 0.92.0 with spanmetrics connector, and the prometheus exporter with setting metric_expiration: 10m

After 10 minutes have passed metrics that where not updated should disappear from the prometheus metric scrapping endpoint.
However, metrics are not disappearing. To test this I instrumented a test application, sent some telemetry, then killed the application. The metrics are never erased

after 20 minutes:

Steps to Reproduce

• Start the collector with following configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  prometheus:
    endpoint: "0.0.0.0:9130"
    metric_expiration: 10m
    add_metric_suffixes: false

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [125ms, 500ms, 2000ms, 8000ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.route
      - name: http.scheme
      - name: http.status_code
      - name: net.host.name
      - name: net.host.port
      - name: net.protocol.name
      - name: net.protocol.version
      - name: net.sock.host.addr
      - name: net.sock.peer.addr

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [spanmetrics]
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheus]

• Send some traces
• Stop sending any telemetry data
• Check the prometheus exporter endpoint: curl -s localhost:9130/metrics | grep duration_count | grep SPAN_KIND_SERVER

Expected Result

After 10 minutes of metrics not being updated, metric_expiration should kick in, and metrics should be removed from prometheus endpoint

Actual Result

Metrics are never removed from prometheus endpoint

Collector version

0.92.0

Environment information

Environment

Ubuntu 22.04.2 LTS

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  prometheus:
    endpoint: "0.0.0.0:9130"
    metric_expiration: 10m
    add_metric_suffixes: false

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [125ms, 500ms, 2000ms, 8000ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.route
      - name: http.scheme
      - name: http.status_code
      - name: net.host.name
      - name: net.host.port
      - name: net.protocol.name
      - name: net.protocol.version
      - name: net.sock.host.addr
      - name: net.sock.peer.addr

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [spanmetrics]
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheus]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for connector/spanmetrics: @albertteoh @portertech. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[toughnoah]

@tqi-raurora Hi, I think it is related to this issue, when you set aggregation_temporality: AGGREGATION_TEMPORALITY_CUMULATIVE, which is the default setting, the connector will keep sending metrics(I have debugged into the source code and I found that metircs comes with new timestamp every time). If you change to AGGREGATION_TEMPORALITY_DELTA, metrics are deleted successfully.

[tqi-raurora]

@toughnoah Hi, thank you, you're right, I think they are caused by the same issue

[swar8080]

Opened #31671 to discuss a possible solution using delta span metrics, but it'd need a code change

[tqi-raurora]

This was fixed by #30559 on release 0.97.0 by adding the metrics_expiration parameter to the spanmetrics connection
I'm closing this issue