[prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend #15281

Dongqi-Guo · 2022-10-19T03:27:29Z

What happened?

Description

I tried to use promethrus remote write to thanos backend and then display metrics on Grafana. I found there are many errors in otel collector log like "Permanent error: invalid temporality and type combination", In result, thanos lacks many metrics used in Grafana dashboard, any idea or solution about this?

Steps to Reproduce

fluentbit node_metrics output
prometheus remote write exporter
thanos as the backend

Expected Result

Actual Result

Collector version

0.61.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

2022-10-19T11:27:07.375+0800    error   exporterhelper/queued_retry.go:395      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination;

Additional context

No response

github-actions · 2022-10-19T19:52:56Z

Pinging code owners: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.

krupyansky · 2022-10-21T11:57:32Z

The problem is reproduced also with VictoriaMetrics as the backend

krupyansky · 2022-10-25T08:18:17Z

Clarification

The error occurs because https://github.com/open-telemetry/opentelemetry-go sends metrics that have 0 DataPoints

Aneurysm9 · 2022-10-31T18:20:03Z

Clarification

The error occurs because https://github.com/open-telemetry/opentelemetry-go sends metrics that have 0 DataPoints

Is there an existing issue on the OTel-Go repo that can be linked here? If not, can you create one with steps to reproduce?

krupyansky · 2022-10-31T18:21:40Z

Exactly @Aneurysm9

I make issue for https://github.com/open-telemetry/opentelemetry-go

open-telemetry/opentelemetry-go#3394

montag · 2022-12-14T06:41:09Z

I'm having the same issue with prometheusremotewrite writing to prometheus. The collector is dropping thousands of metrics.

2022-12-14T06:07:04.831Z error exporterhelper/queued_retry.go:394 Exporting failed. The error is not retryable. Dropping data. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “prometheusremotewrite”, “error”: “Permanent error: invalid temporality and type combination”, “dropped_items”: 20}

krupyansky · 2022-12-14T09:22:53Z

@montag do you use open-telemetry library https://github.com/open-telemetry/opentelemetry-go?

Try update version to latest
https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.11.2

montag · 2022-12-14T16:01:35Z

Thanks, @krupyansky. I'm using the otel collector helm chart via terraform.

krupyansky · 2022-12-14T16:08:24Z

@montag are you sending metrics from your application to otel collectore via https://github.com/open-telemetry/opentelemetry-go?

montag · 2022-12-14T16:30:16Z

@krupyansky I'm using the python open-telemetry instrumentation libs to send to the otel collector (otlp receiver), which then uses the prometheusremotewrite exporter to push to prom. I see the above error in the collector logs every few minutes.

krupyansky · 2022-12-14T17:09:14Z

@montag try to write issue to the python open-telemetry instrumentation libs like my issue open-telemetry/opentelemetry-go#3394

Most likely the error occurs in your case because the python open-telemetry sends metrics that have 0 DataPoints

montag · 2022-12-14T21:02:58Z

@krupyansky Any idea how I might verify that?

edawader · 2023-02-10T22:48:24Z

In same boat as @montag - No idea how we could be sending metrics with 0 Datapoints.

GorodentsevD · 2023-03-29T13:52:50Z

Hi! Could you please clarify, do we have some showstoppers to upgrade opentelemetry-go version with the fix? Got the same bug on the 0.74.0 version of opentelemetry-contrib. And if i am correct, it seems that the collector is using v1.0.0-rc8 opentelemetry-go https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheusremotewrite/go.mod#L11. And version with the fix starts from v1.11.2

Aneurysm9 · 2023-04-05T16:53:44Z

Hi! Could you please clarify, do we have some showstoppers to upgrade opentelemetry-go version with the fix? Got the same bug on the 0.74.0 version of opentelemetry-contrib. And if i am correct, it seems that the collector is using v1.0.0-rc8 opentelemetry-go https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/translator/prometheusremotewrite/go.mod#L11. And version with the fix starts from v1.11.2

That dependency is on the pdata module from the collector core repository, not on the OTel Go SDK. The issue here is with data produced by an application using the OTel Go SDK and not any use of the SDK within the collector framework or components.

GorodentsevD · 2023-04-05T17:00:52Z

Thank you for the answer! Sorry, misunderstand the discussion, now i get it. I have the same issue with Kong statsd plugin

github-actions · 2023-06-05T03:30:29Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

eugene-chernyshenko · 2023-07-24T21:09:34Z

I have the same issue with opentelemetry-collector-contrib 0.81.0

2023-07-24T21:17:44.155Z	info	service/service.go:148	Everything is ready. Begin running and processing data.
2023-07-24T21:18:37.504Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 2}
2023-07-24T21:18:37.504Z	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.19.0)
     -> namespace: Str(develop)
     -> service.name: Str(scrape)
     -> telemetry.auto.version: Str(0.40b0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope opentelemetry.instrumentation.flask 0.40b0
Metric #0
Descriptor:
     -> Name: http.server.active_requests
     -> Description: measures the number of concurrent HTTP requests that are currently in-flight
     -> Unit: requests
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> http.host: Str(127.0.0.1:5000)
     -> http.scheme: Str(http)
     -> http.flavor: Str(1.1)
     -> http.server_name: Str(0.0.0.0)
StartTimestamp: 2023-07-24 20:43:23.343899867 +0000 UTC
Timestamp: 2023-07-24 21:18:37.404773474 +0000 UTC
Value: 0
Metric #1
Descriptor:
     -> Name: http.server.duration
     -> Description: measures the duration of the inbound HTTP request
     -> Unit: ms
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> http.host: Str(127.0.0.1:5000)
     -> http.scheme: Str(http)
     -> http.flavor: Str(1.1)
     -> http.server_name: Str(0.0.0.0)
     -> net.host.port: Int(5000)
     -> http.status_code: Int(200)
StartTimestamp: 2023-07-24 20:43:23.346501178 +0000 UTC
Timestamp: 2023-07-24 21:18:37.404773474 +0000 UTC
Count: 7
Sum: 16.000000
Min: 1.000000
Max: 3.000000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 7
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-07-24T21:19:37.422Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 1}
2023-07-24T21:19:37.422Z	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.19.0)
     -> namespace: Str(develop)
     -> service.name: Str(scrape)
     -> telemetry.auto.version: Str(0.40b0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope opentelemetry.instrumentation.flask 0.40b0
Metric #0
Descriptor:
     -> Name: http.server.active_requests
     -> Description: measures the number of concurrent HTTP requests that are currently in-flight
     -> Unit: requests
     -> DataType: Sum
     -> IsMonotonic: false
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> http.method: Str(GET)
     -> http.host: Str(127.0.0.1:5000)
     -> http.scheme: Str(http)
     -> http.flavor: Str(1.1)
     -> http.server_name: Str(0.0.0.0)
StartTimestamp: 2023-07-24 20:43:23.343899867 +0000 UTC
Timestamp: 2023-07-24 21:19:37.407265471 +0000 UTC
Value: 0
Metric #1
Descriptor:
     -> Name: http.server.duration
     -> Description: measures the duration of the inbound HTTP request
     -> Unit: ms
     -> DataType: Empty
	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-07-24T21:19:37.423Z	error	exporterhelper/queued_retry.go:391	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination for metric \"http.server.duration\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
	go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
	go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
	go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
	go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/internal/bounded_memory_queue.go:47

github-actions · 2023-10-16T03:30:11Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

ElementalWarrior · 2023-11-29T08:08:51Z

We were experiencing this for the python metrics instrumentation.

You can resolve it with a filter processor:

processors:
  filter/empty_http_server_duration:
    error_mode: ignore
    metrics:
      metric:
          - 'name == "http.server.duration" and type != METRIC_DATA_TYPE_HISTOGRAM'
          - 'name == "http.client.duration" and type != METRIC_DATA_TYPE_HISTOGRAM'

github-actions · 2024-01-30T03:29:06Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

francescopotenziani · 2024-02-27T10:53:24Z

Any updates? I've the same issue of @montag: I'm using:

otel/opentelemetry-collector-contrib:0.67.0
"@opentelemetry/sdk-node": "0.33.0"

Starefossen · 2024-03-05T07:49:19Z

Getting the same with the most recent version of the collector:

2024-03-05T07:46:50.226Z        error   exporterhelper/common.go:95     Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination for metric \"app_currency_counter\"", "dropped_items": 1}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter.createMetricsExporter.WithQueue.func2.1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/common.go:95
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/queue_sender.go:117
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue[...]).Consume
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/internal/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*QueueConsumers[...]).Start.func1
        go.opentelemetry.io/collector/exporter@v0.93.0/exporterhelper/internal/consumers.go:43

…lure to translate metrics (#29729) Don't drop a whole batch in case of failure to translate from Otel to Prometheus. Instead, with this PR we are trying to send to Prometheus all the metrics that were properly translated and create a warning message for failures to translate. This PR also adds supports to telemetry in this component so that it is possible to inspect how the translation process is happening and identify failed translations. I opted to not include the number of time series that failed translation because I don't want to make assumptions about how the `FromMetrics` function works. Instead we are just publishing if there was any failure during the translation process and the number of time series returned. **Link to tracking Issue:** #15281 **Testing:** UTs were added to account for the case that you have mixed metrics, with some succeeding the translation and some failing. --------- Signed-off-by: Raphael Silva <rapphil@gmail.com> Co-authored-by: Anthony Mirabella <a9@aneurysm9.com> Co-authored-by: bryan-aguilar <46550959+bryan-aguilar@users.noreply.github.com> Co-authored-by: Bryan Aguilar <bryaag@amazon.com>

github-actions · 2024-05-06T03:29:44Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-07-05T05:19:19Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

Dongqi-Guo added bug Something isn't working needs triage New item requiring triage labels Oct 19, 2022

evan-bradley added priority:p2 Medium exporter/prometheusremotewrite and removed needs triage New item requiring triage labels Oct 19, 2022

github-actions bot added the Stale label Jun 5, 2023

github-actions bot removed the Stale label Jul 25, 2023

github-actions bot added the Stale label Oct 16, 2023

github-actions bot removed the Stale label Nov 30, 2023

rapphil mentioned this issue Dec 11, 2023

[exporter/prometheusremotewrite] Fix: Don't drop batch in case of failure to translate metrics #29729

Merged

crobert-1 mentioned this issue Dec 19, 2023

[prometheusremotewrite] exporter complaining about temporality in OTEL pipeline #30094

Closed

jmichalek132 mentioned this issue Jan 14, 2024

new component: deltatocumulative processor #30479

Closed

edeNFed mentioned this issue Jan 22, 2024

Permanent error: invalid temporality and type combination; invalid temporality and type combination odigos-io/odigos#792

Closed

github-actions bot added the Stale label Jan 30, 2024

github-actions bot removed the Stale label Feb 28, 2024

Starefossen mentioned this issue Mar 5, 2024

OpenTelemetry Collector is dropping invalid metric "app_currency_counter" open-telemetry/opentelemetry-demo#1432

Closed

github-actions bot added the Stale label May 6, 2024

github-actions bot added the closed as inactive label Jul 5, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend #15281

[prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend #15281

Dongqi-Guo commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

krupyansky commented Oct 21, 2022

krupyansky commented Oct 25, 2022

Aneurysm9 commented Oct 31, 2022

krupyansky commented Oct 31, 2022 •

edited

Loading

montag commented Dec 14, 2022

krupyansky commented Dec 14, 2022

montag commented Dec 14, 2022

krupyansky commented Dec 14, 2022 •

edited

Loading

montag commented Dec 14, 2022 •

edited

Loading

krupyansky commented Dec 14, 2022

montag commented Dec 14, 2022

edawader commented Feb 10, 2023

GorodentsevD commented Mar 29, 2023 •

edited

Loading

Aneurysm9 commented Apr 5, 2023

GorodentsevD commented Apr 5, 2023

github-actions bot commented Jun 5, 2023

eugene-chernyshenko commented Jul 24, 2023 •

edited

Loading

github-actions bot commented Oct 16, 2023

ElementalWarrior commented Nov 29, 2023

github-actions bot commented Jan 30, 2024

francescopotenziani commented Feb 27, 2024

Starefossen commented Mar 5, 2024

github-actions bot commented May 6, 2024

github-actions bot commented Jul 5, 2024

[prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend #15281

[prometheusremotewrite] invalid temporality and type combination when remote write to thanos backend #15281

Comments

Dongqi-Guo commented Oct 19, 2022

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Oct 19, 2022

krupyansky commented Oct 21, 2022

krupyansky commented Oct 25, 2022

Aneurysm9 commented Oct 31, 2022

krupyansky commented Oct 31, 2022 • edited Loading

montag commented Dec 14, 2022

krupyansky commented Dec 14, 2022

montag commented Dec 14, 2022

krupyansky commented Dec 14, 2022 • edited Loading

montag commented Dec 14, 2022 • edited Loading

krupyansky commented Dec 14, 2022

montag commented Dec 14, 2022

edawader commented Feb 10, 2023

GorodentsevD commented Mar 29, 2023 • edited Loading

Aneurysm9 commented Apr 5, 2023

GorodentsevD commented Apr 5, 2023

github-actions bot commented Jun 5, 2023

eugene-chernyshenko commented Jul 24, 2023 • edited Loading

github-actions bot commented Oct 16, 2023

ElementalWarrior commented Nov 29, 2023

github-actions bot commented Jan 30, 2024

francescopotenziani commented Feb 27, 2024

Starefossen commented Mar 5, 2024

github-actions bot commented May 6, 2024

github-actions bot commented Jul 5, 2024

krupyansky commented Oct 31, 2022 •

edited

Loading

krupyansky commented Dec 14, 2022 •

edited

Loading

montag commented Dec 14, 2022 •

edited

Loading

GorodentsevD commented Mar 29, 2023 •

edited

Loading

eugene-chernyshenko commented Jul 24, 2023 •

edited

Loading