[Prometheus Compliance Tests] TestRemoteWrite/otelcollector/Counter failed on main

[songy23]

Component(s)

No response

Describe the issue you're reporting

E.g. https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/10791551495/job/29929199424?pr=35112

=== NAME  TestRemoteWrite/otelcollector/Counter
    counter.go:35: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/compliance/remotewrite/sender/cases/counter.go:35
        	            				/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/compliance/remotewrite/sender/main_test.go:108
        	            				/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/compliance/remotewrite/sender/main_test.go:72
        	Error:      	Should be true
        	Test:       	TestRemoteWrite/otelcollector/Counter
        	Messages:   	found zero samples for {__name__="counter"}

[VihasMakwana]

Not able to reproduce this on my darwin arm64...

Same for linux amd64

With some effort, I'm able to reproduce this. Will try to track down and fix it.

[atoulme]

This is breaking on main and all PRs.

[TylerHelmuth]

@dashpole can you take a look at this?

[dashpole]

I'll try to take a look. cc @jmichalek132 @ArthurSens since this is related to the PRW exporter.

[dmitryax]

Marking as a release blocker so we don't release with broken CI

[dashpole]

Tested against older versions of the prometheus/compliance repo and confirmed it was not a recent regression there

[dashpole]

In this run, the test actually failed, but the compliance test step passed: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/10495822874/job/29075112350

[dashpole]

So prior to #35071, the test was failing, but the step itself was not failing. The PR added set -o pipefail && \, which seems to have made the failure fail the test.

[dashpole]

I took a wild guess and was correct! The compliance test needs to set add_metric_suffixes: false on the prometheusremotewrite exporter for tests to pass. I assume the counter was being renamed to counter_total...

[dashpole]

prometheus/compliance#106 fixes tests for me locally

[ArthurSens]

Merged, could we try again?

[djaglowski]

I updated #35026 and it worked there! Thanks @dashpole for getting to the bottom of this.

[crobert-1]

Thanks again @dashpole, really appreciate the fix!