[exporter/signalfx] TestCorrelationClient unit test failure

[crobert-1]

Component(s)

exporter/signalfx

Describe the issue you're reporting

CI action failure

--- FAIL: TestCorrelationClient (10.03s)
    client_test.go:61: Test server got PUT request: /v2/apm/correlate/host/test-box/service
    client_test.go:61: Test server got DELETE request: /v2/apm/correlate/host/test-box/service/test-service
    client_test.go:61: Test server got PUT request: /v2/apm/correlate/host/test-box/environment
    client_test.go:61: Test server got DELETE request: /v2/apm/correlate/host/test-box/environment/test-service
    client_test.go:61: Test server got GET request: /v2/apm/correlate/host/test-box
    --- FAIL: TestCorrelationClient/does_retry_500_responses (4.01s)
        client_test.go:250: 
            	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/exporter/signalfxexporter/internal/apm/correlations/client_test.go:250
            	Error:      	Not equal: 
            	            	expected: 5
            	            	actual  : 10
            	Test:       	TestCorrelationClient/does_retry_500_responses
FAIL
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/signalfxexporter/internal/apm/correlations

[djaglowski]

A couple more occurences:

• https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6267478343/job/17022895003#step:9:451
• https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/6275525361/job/17043289351?pr=27075#step:9:451

[crobert-1]

I'm looking into this. I can't reproduce locally and it's only failing sometimes so it seems environment dependent somehow. I did find during debugging that the actual response count can end up being much lower if the program is paused during the timeout window, so there's at least some way to change resulting behavior here.

[crobert-1]

Adding more context: This test was introduced with the deprecation of the Signalfx Agent. A lot of code was moved over for #27040, which was merged the day before this issue was filed.

[crobert-1]

I believe I've found the issue here:

			CleanupInterval: 0,

The test sets up a client for correlating dimensions on data. Deduplication is done on the client side as a performance enhancement to make fewer update requests to the server. Deduplication on data is done within a given time context, so the client is deduplicating any data seen within the CleanupInterval (essentially). When it's set to 0 here it's making it so that there's almost no chance to dedup data, since there will likely be an event to trigger purging the requests before there can be multiple correlation calls. This results in no deduplication being done and the test failing.