Flaky test: TestServiceTelemetryReusable

[codeboten]

See the failure: https://github.com/open-telemetry/opentelemetry-collector/actions/runs/3362867290/jobs/5575186303#step:6:4285

 2022-10-31T16:16:23.033Z	info	service/service.go:106	Everything is ready. Begin running and processing data.
    service_test.go:179: 
        	Error Trace:	/home/runner/work/opentelemetry-collector/opentelemetry-collector/service/service_test.go:179
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:8888/metrics": dial tcp [::1]:8888: connect: connection refused
        	Test:       	TestServiceTelemetryReusable
--- FAIL: TestServiceTelemetryReusable (0.01s)

[sakshi1215]

@codeboten should we add a retry mechanism during the connection? I think the server might not be ready which might have caused this, lmk your opinions.

[codeboten]

@sakshi1215 it's possible a retry would solve this, would be good to confirm that's the root cause first. Another thing that may be causing issues is if the port is being re-used by multiple tests.

[paivagustavo]

Just hit this in #6423 as well. I could recreate it by running this test multiple times with -test.count=100. And it does seems that the server was not ready yet (it gets started in a go routine, so very possible), we could possibly use assert.Eventually here and check if we can still recreate this failure.

Also agree that we should avoid using a static port here and instead getting an available one.