Flaky test TestOutputReload

[jsoriano]

Flaky Test

• Test Name: TestOutputReload
• Link:

  beats/libbeat/publisher/pipeline/controller_test.go

  Line 39 in 7028e2c

  func TestOutputReload(t *testing.T) {

• Branch: Seen in Bump version to 7.11.0 #21629, but also in other PRs.
• Notes: Can be reproduced locally with go test -count=..., seems to be a possible goroutines leak.

Stack Trace

Seen in 7.x:

....
....
goroutine 111033 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*clientWorker).run(0xc000616040)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:96 +0xb8
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194

goroutine 111125 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*clientWorker).run(0xc000488280)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:96 +0xb8
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194
--- FAIL: TestOutputReload (8.15s)
    --- FAIL: TestOutputReload/client (6.58s)
        util.go:39: reproduce test with `go test ... -seed 1602089589034742949`
        controller_test.go:113: Possible goroutines leak, before: 233, after: 234

Seen in master:

--- FAIL: TestOutputReload (7.70s)
    --- FAIL: TestOutputReload/network_client (6.37s)
        util.go:39: reproduce test with `go test ... -seed 1602091446357947486`
        controller_test.go:113: Possible goroutines leak, before: 5, after: 6
goroutine 396459 [running]:
runtime/pprof.writeGoroutineStacks(0x10c0120, 0xc0001a2008, 0xc000064d30, 0xdba795)
	/home/jaime/go/src/runtime/pprof/pprof.go:665 +0x9d
runtime/pprof.writeGoroutine(0x10c0120, 0xc0001a2008, 0x2, 0x15b01a8, 0x1812180)
	/home/jaime/go/src/runtime/pprof/pprof.go:654 +0x44
runtime/pprof.(*Profile).WriteTo(0x17f7580, 0x10c0120, 0xc0001a2008, 0x2, 0xf2c7b9ab8, 0x198ac00)
	/home/jaime/go/src/runtime/pprof/pprof.go:329 +0x3da
github.com/elastic/beats/v7/libbeat/tests/resources.GoroutinesChecker.check(0x5, 0x12a05f200, 0x10df160, 0xc00024c000, 0x20, 0xc0001b0220)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/tests/resources/goroutines.go:69 +0x12d
github.com/elastic/beats/v7/libbeat/tests/resources.GoroutinesChecker.Check(0x5, 0x12a05f200, 0x10df160, 0xc00024c000)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/tests/resources/goroutines.go:52 +0x61
github.com/elastic/beats/v7/libbeat/publisher/pipeline.TestOutputReload.func1(0xc00024c000)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/controller_test.go:113 +0x1c8
testing.tRunner(0xc00024c000, 0xc0004aa000)
	/home/jaime/go/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
	/home/jaime/go/src/testing/testing.go:1042 +0x357

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00024c360, 0xfb7965, 0x10, 0xfdec58, 0x7ca201)
	/home/jaime/go/src/testing/testing.go:1043 +0x37e
testing.runTests.func1(0xc00024c120)
	/home/jaime/go/src/testing/testing.go:1284 +0x78
testing.tRunner(0xc00024c120, 0xc000122e10)
	/home/jaime/go/src/testing/testing.go:991 +0xdc
testing.runTests(0xc00000e140, 0x17fc360, 0x6, 0x6, 0x0)
	/home/jaime/go/src/testing/testing.go:1282 +0x2a7
testing.(*M).Run(0xc0001f4000, 0x0)
	/home/jaime/go/src/testing/testing.go:1199 +0x15f
main.main()
	_testmain.go:54 +0x135

goroutine 51 [select]:
go.elastic.co/apm/transport/transporttest.ErrorTransport.SendStream(0x0, 0x0, 0x10d4cc0, 0xc0005aa380, 0x10bfb40, 0xc0003c5d40, 0xbaa601, 0x101)
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/transport/transporttest/err.go:45 +0x123
go.elastic.co/apm.(*Tracer).loop.func2(0xc000182780, 0xc000182720, 0x10d4cc0, 0xc0005aa380, 0xc0001a5320, 0xc000186480, 0xc0001a2078)
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/tracer.go:783 +0x1d0
created by go.elastic.co/apm.(*Tracer).loop
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/tracer.go:773 +0x36e

goroutine 35 [select]:
go.elastic.co/apm.(*Tracer).loop(0xc000186480)
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/tracer.go:873 +0xb28
created by go.elastic.co/apm.newTracer
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/tracer.go:433 +0x7cc

goroutine 385516 [chan receive]:
testing.(*T).Run(0xc00024c000, 0xfb6a7e, 0xe, 0xc0004aa000, 0xc000480f01)
	/home/jaime/go/src/testing/testing.go:1043 +0x37e
github.com/elastic/beats/v7/libbeat/publisher/pipeline.TestOutputReload(0xc00024c360)
	/home/jaime/gocode/src/github.com/elastic/beats/libbeat/publisher/pipeline/controller_test.go:46 +0x19f
testing.tRunner(0xc00024c360, 0xfdec58)
	/home/jaime/go/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
	/home/jaime/go/src/testing/testing.go:1042 +0x357

goroutine 406191 [chan receive]:
go.elastic.co/apm/internal/iochan.(*Reader).Read(0xc0003c5d40, 0xc0002c8000, 0x2000, 0x2000, 0x2cd, 0x0, 0x0)
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/internal/iochan/reader.go:87 +0x135
io/ioutil.devNull.ReadFrom(0x0, 0x10bfb40, 0xc0003c5d40, 0xed9f40, 0x1, 0x7fdb797c4170)
	/home/jaime/go/src/io/ioutil/ioutil.go:144 +0x92
io.copyBuffer(0x10c1a60, 0x19b6800, 0x10bfb40, 0xc0003c5d40, 0x0, 0x0, 0x0, 0x0, 0x77d336, 0xfdfa38)
	/home/jaime/go/src/io/io.go:391 +0x2fc
io.Copy(...)
	/home/jaime/go/src/io/io.go:364
go.elastic.co/apm/transport/transporttest.ErrorTransport.SendStream.func1(0x10bfb40, 0xc0003c5d40, 0xc0005ea540)
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/transport/transporttest/err.go:42 +0x67
created by go.elastic.co/apm/transport/transporttest.ErrorTransport.SendStream
	/home/jaime/gocode/pkg/mod/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/transport/transporttest/err.go:41 +0x89
--- FAIL: TestOutputReload (8.14s)
    --- FAIL: TestOutputReload/network_client (6.62s)
        util.go:39: reproduce test with `go test ... -seed 1602091475354293056`
        controller_test.go:113: Possible goroutines leak, before: 5, after: 6
FAIL
FAIL	github.com/elastic/beats/v7/libbeat/publisher/pipeline	68.189s
FAIL

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlind23]

Backlog grooming: Closing for now.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)