[Flaky Test]: TestBeatsReceiverLogs – Error: Condition never satisfied

[pchila]

Failing test case

TestBeatsReceiverLogs

Error message

A series of asserts is failing regarding beats receivers logs is failing, leaving an elastic agent process running

Build

https://buildkite.com/elastic/elastic-agent/builds/26642#01993909-3d3d-42d4-a960-c6046ee71593/319-691

OS

Linux

Stacktrace and notes

2025-09-11 14:26:32 UTC
	    beat_receivers_test.go:734:
2025-09-11 14:26:32 UTC
	        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:810
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:738
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.7/go/src/runtime/asm_arm64.s:1223
2025-09-11 14:26:32 UTC
	        	Error:      	Not equal:
2025-09-11 14:26:32 UTC
	        	            	expected: 2
2025-09-11 14:26:32 UTC
	        	            	actual  : 0
2025-09-11 14:26:32 UTC
	    beat_receivers_test.go:734:
2025-09-11 14:26:32 UTC
	        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:811
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:738
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.7/go/src/runtime/asm_arm64.s:1223
2025-09-11 14:26:32 UTC
	        	Error:      	Not equal:
2025-09-11 14:26:32 UTC
	        	            	expected: "beat-v2-client"
2025-09-11 14:26:32 UTC
	        	            	actual  : ""
2025-09-11 14:26:32 UTC
	
2025-09-11 14:26:32 UTC
	        	            	Diff:
2025-09-11 14:26:32 UTC
	        	            	--- Expected
2025-09-11 14:26:32 UTC
	        	            	+++ Actual
2025-09-11 14:26:32 UTC
	        	            	@@ -1 +1 @@
2025-09-11 14:26:32 UTC
	        	            	-beat-v2-client
2025-09-11 14:26:32 UTC
	        	            	+
2025-09-11 14:26:32 UTC
	    beat_receivers_test.go:734:
2025-09-11 14:26:32 UTC
	        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:813
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:738
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.7/go/src/runtime/asm_arm64.s:1223
2025-09-11 14:26:32 UTC
	        	Error:      	Not equal:
2025-09-11 14:26:32 UTC
	        	            	expected: 2
2025-09-11 14:26:32 UTC
	        	            	actual  : 0
2025-09-11 14:26:32 UTC
	    beat_receivers_test.go:734:
2025-09-11 14:26:32 UTC
	        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:813
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:738
2025-09-11 14:26:32 UTC
	        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.7/go/src/runtime/asm_arm64.s:1223
2025-09-11 14:26:32 UTC
	        	Error:      	Not equal:
2025-09-11 14:26:32 UTC
	        	            	expected: 2
2025-09-11 14:26:32 UTC
	        	            	actual  : 0
2025-09-11 14:26:32 UTC
	    beat_receivers_test.go:734:
2025-09-11 14:26:32 UTC
	        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1757598785638537982/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:734
2025-09-11 14:26:32 UTC
	        	Error:      	Condition never satisfied
2025-09-11 14:26:32 UTC
	        	Test:       	TestBeatsReceiverLogs
2025-09-11 14:26:32 UTC
	    fixture.go:1427: Temporary directory "/tmp/TestBeatsReceiverLogs3333832855" preserved for investigation/debugging

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pchila]

I included the run on my PR but the issue seems to occur also on main, some examples:

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/10094#019933a6-204b-40a0-80da-44913398c8ac/185-520
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/10094#019933a6-2046-414c-b617-5ef9ee5fbaba/189-522
https://buildkite.com/elastic/elastic-agent-extended-testing/builds/10092#019932bb-ca5c-49e2-9e70-5bb098abe039/188-456

[pkoutsovasilis]

Thanks for creating this @pchila, here is another occurrence https://buildkite.com/elastic/elastic-agent/builds/26644#01993919-9f24-4a1a-ad97-bb38076cf322/249-584

[swiatekm]

I got this test to fail with improved diagnostics here: https://buildkite.com/elastic/elastic-agent/builds/27545#0199820a-0430-4456-8463-59c4b5dba014

The actual error is OTel manager failed: failed to build pipelines: failed to create "metricbeatreceiver/_agent-component/system/metrics-default" receiver for data type "logs": error creating metricbeatreceiver: could not start the HTTP server for the API: failed to listen on the named pipe \\.\pipe\iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: open \\.\pipe\iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: Access is denied..

Does anyone know why we'd get a permission error when creating a named pipe on Windows? It seems consistent in that rerunning the test on the same buildkite runner gives the same result.

[swiatekm]

https://github.com/elastic/ingest-dev/issues/6133 appears to resolve the problem. I'm going to rewrite this test to be easier to troubleshoot as well in #10153.