[test][exporter/prometheus] TestPrometheusExporter failures

[pjanotti]

Component(s)

exporter/prometheus

Describe the issue you're reporting

A few instances of failures:

• https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/11616181947/job/32348529258#step:8:54
• https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/11633600629/job/32399072258#step:8:282

=== FAIL: . TestPrometheusExporter (0.00s)
    prometheus_test.go:87: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/exporter/prometheusexporter/prometheus_test.go:87
        	Error:      	Received unexpected error:
        	            	close tcp 127.0.0.1:8999: use of closed network connection
        	Test:       	TestPrometheusExporter

=== FAIL: . TestPrometheusExporter (re-run 1) (0.01s)
    prometheus_test.go:87: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/exporter/prometheusexporter/prometheus_test.go:87
        	Error:      	Received unexpected error:
        	            	close tcp 127.0.0.1:8999: use of closed network connection
        	Test:       	TestPrometheusExporter

DONE 2 runs, 73 tests, 2 failures in 361.711s

In quick glance this seems likely because the server using the port is actually launched in a goroutine and the port can be closed by shutdown before the server starts.

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

@Argannor this seems likely. to be related to #35465.

I'm OOO for the next few weeks. Are you able to investigate + fix or alternatively revert your PR?

Thanks!

[Argannor]

@dashpole I can have a closer look on monday and come up with a new PR. If that's not fast enough for you feel free to revert and I'll open a new PR reintroducing the changes + fix.

In quick glance this seems likely because the server using the port is actually launched in a goroutine and the port can be closed by shutdown before the server starts.

I think that's a likely explanation. I'll do some experimentation to confirm your hypothesis. If I'm not mistaken the error only occurs sporadically, so it's likely to be a race condition.

My apologies.

[dashpole]

Monday should be fine. I won't be able to review for a while, but other approvers are free to do so.

[Argannor]

I can confirm that this error happens when srv.Serve(ln) is called after ln.Close() has already been called. Will follow up with PR.

Edit: While this sequence causes an error like this, I don't think this is the root cause, since the error returned by srv.Serve(ln) is ignored, and the test fails during a call to prometheusExporter.Shutdown().

[pjanotti]

@Argannor correct, that first glance was wrong. Looking again at the error message it says close tcp 127.0.0.1:8999: use of closed network connection the first "close" confirms the operation attempted, so as you saw it was on shutdown.