Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metricbeat/module/mongodb/replstatus.TestFetch Go integration test is flaky #30610

Closed
cmacknz opened this issue Feb 28, 2022 · 2 comments
Closed

metricbeat/module/mongodb/replstatus.TestFetch Go integration test is flaky #30610

cmacknz opened this issue Feb 28, 2022 · 2 comments
Labels
flaky-test Unstable or unreliable test cases. Team:Integrations Label for the Integrations team

Comments

@cmacknz
Copy link
Member

cmacknz commented Feb 28, 2022

Flaky Test

Three consecutive builds of the main branch had this test fail today:

  1. https://beats-ci.elastic.co/job/Beats/job/beats/job/main/129/
  2. https://beats-ci.elastic.co/job/Beats/job/beats/job/main/130/
  3. https://beats-ci.elastic.co/job/Beats/job/beats/job/main/131/
  • Test Name: metricbeat-goIntegTest / github.com/elastic/beats/v7/metricbeat/module/mongodb/replstatus.TestFetch
  • Link: Link to file/line number in github.
  • Branch: main
  • Notes: The tests only stops because is hits the 10 minute timeout set by the go test invocation. The test is not hitting an error or an internal timeout, it appears to be stuck.

Stack Trace

[2022-02-28T16:48:58.248Z] === FAIL: metricbeat/module/mongodb/replstatus TestFetch (unknown)
...
[2022-02-28T16:18:33.349Z]     replstatus_integration_test.go:124: Mongodb state is 0
[2022-02-28T16:18:33.349Z]     replstatus_integration_test.go:124: Mongodb state is 0
[2022-02-28T16:18:33.349Z]     replstatus_integration_test.go:124: Mongodb state is 0
[2022-02-28T16:18:33.349Z]     replstatus_integration_test.go:124: Mongodb state is 0
[2022-02-28T16:18:33.349Z] coverage: 1.0% of statements
[2022-02-28T16:18:33.349Z] panic: test timed out after 10m0s
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 41 [running]:
[2022-02-28T16:18:33.349Z] testing.(*M).startAlarm.func1()
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1788 +0x8e
[2022-02-28T16:18:33.349Z] created by time.goFunc
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/time/sleep.go:180 +0x31
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 1 [chan receive, 10 minutes]:
[2022-02-28T16:18:33.349Z] testing.(*T).Run(0xc0003c8680, {0xfe66a1, 0x787333}, 0x10329b0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1307 +0x375
[2022-02-28T16:18:33.349Z] testing.runTests.func1(0xc000262bd0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1598 +0x6e
[2022-02-28T16:18:33.349Z] testing.tRunner(0xc0003c8680, 0xc00063fce0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1259 +0x102
[2022-02-28T16:18:33.349Z] testing.runTests(0xc0005b7f00, {0x15e74a0, 0x2, 0x2}, {0xeeff40, 0xc000115cb0, 0x1657600})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1596 +0x43f
[2022-02-28T16:18:33.349Z] testing.(*M).Run(0xc0005b7f00)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1504 +0x51d
[2022-02-28T16:18:33.349Z] main.main()
[2022-02-28T16:18:33.349Z] 	_testmain.go:99 +0x1f5
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 22 [runnable]:
[2022-02-28T16:18:33.349Z] sync.runtime_SemacquireMutex(0xfc3760, 0x80, 0xeb5c00)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/runtime/sema.go:71 +0x25
[2022-02-28T16:18:33.349Z] sync.(*Mutex).lockSlow(0xc00055a508)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/sync/mutex.go:138 +0x165
[2022-02-28T16:18:33.349Z] sync.(*Mutex).Lock(...)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/sync/mutex.go:81
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*mongoSocket).SimpleQuery(0x3b, 0xc0003ba540)
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:367 +0x1d4
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*Database).run(0xc0002c5dc0, 0x101, {0xf08260, 0xc0000a15c0}, {0xe9a000, 0xc0000100e0})
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/session.go:3261 +0x26e
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*Database).Run(0xc0002c5dc0, {0xf08260, 0xc0000a15c0}, {0xe9a000, 0xc0000100e0})
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/session.go:656 +0xd6
[2022-02-28T16:18:33.349Z] github.com/elastic/beats/v7/metricbeat/module/mongodb/replstatus.initiateReplicaSet(0xc0003c8820, {0xc0003ee520, 0xfe4ade})
[2022-02-28T16:18:33.349Z] 	/go/src/github.com/elastic/beats/metricbeat/module/mongodb/replstatus/replstatus_integration_test.go:122 +0x3e5
[2022-02-28T16:18:33.349Z] github.com/elastic/beats/v7/metricbeat/module/mongodb/replstatus.TestFetch(0xc0003c8820)
[2022-02-28T16:18:33.349Z] 	/go/src/github.com/elastic/beats/metricbeat/module/mongodb/replstatus/replstatus_integration_test.go:40 +0x7d
[2022-02-28T16:18:33.349Z] testing.tRunner(0xc0003c8820, 0x10329b0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1259 +0x102
[2022-02-28T16:18:33.349Z] created by testing.(*T).Run
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/testing/testing.go:1306 +0x35a
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 33 [IO wait]:
[2022-02-28T16:18:33.349Z] internal/poll.runtime_pollWait(0x7f00ba3837b8, 0x72)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/runtime/netpoll.go:234 +0x89
[2022-02-28T16:18:33.349Z] internal/poll.(*pollDesc).wait(0xc00016c080, 0xc000118de0, 0x0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
[2022-02-28T16:18:33.349Z] internal/poll.(*pollDesc).waitRead(...)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
[2022-02-28T16:18:33.349Z] internal/poll.(*FD).Read(0xc00016c080, {0xc000118de0, 0x24, 0x24})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
[2022-02-28T16:18:33.349Z] net.(*netFD).Read(0xc00016c080, {0xc000118de0, 0x0, 0x0})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/net/fd_posix.go:56 +0x29
[2022-02-28T16:18:33.349Z] net.(*conn).Read(0xc000010090, {0xc000118de0, 0x0, 0xb3ab6c})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/net/net.go:183 +0x45
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.fill({0x111ea88, 0xc000010090}, {0xc000118de0, 0x24, 0xc0005c8d50})
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:535 +0x43
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc00029bdc0)
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:551 +0xab
[2022-02-28T16:18:33.349Z] created by gopkg.in/mgo%2ev2.newSocket
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:194 +0x1ef
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 28 [select]:
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*mongoCluster).syncServersLoop(0xc0003bc000)
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/cluster.go:394 +0x259
[2022-02-28T16:18:33.349Z] created by gopkg.in/mgo%2ev2.newCluster
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/cluster.go:78 +0x1e5
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 30 [sleep]:
[2022-02-28T16:18:33.349Z] time.Sleep(0x37e11d600)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/runtime/time.go:193 +0x12e
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*mongoServer).pinger(0xc0003ba2a0, 0x1)
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/server.go:301 +0x116
[2022-02-28T16:18:33.349Z] created by gopkg.in/mgo%2ev2.newServer
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/server.go:89 +0x186
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] goroutine 32 [IO wait]:
[2022-02-28T16:18:33.349Z] internal/poll.runtime_pollWait(0x7f00ba3838a0, 0x72)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/runtime/netpoll.go:234 +0x89
[2022-02-28T16:18:33.349Z] internal/poll.(*pollDesc).wait(0xc0004fe180, 0xc000042210, 0x0)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
[2022-02-28T16:18:33.349Z] internal/poll.(*pollDesc).waitRead(...)
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
[2022-02-28T16:18:33.349Z] internal/poll.(*FD).Read(0xc0004fe180, {0xc000042210, 0x24, 0x24})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
[2022-02-28T16:18:33.349Z] net.(*netFD).Read(0xc0004fe180, {0xc000042210, 0x0, 0x0})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/net/fd_posix.go:56 +0x29
[2022-02-28T16:18:33.349Z] net.(*conn).Read(0xc0003f6038, {0xc000042210, 0x0, 0xb3ab6c})
[2022-02-28T16:18:33.349Z] 	/usr/local/go/src/net/net.go:183 +0x45
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.fill({0x111ea88, 0xc0003f6038}, {0xc000042210, 0x24, 0xc00014cb00})
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:535 +0x43
[2022-02-28T16:18:33.349Z] gopkg.in/mgo%2ev2.(*mongoSocket).readLoop(0xc0003ba380)
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:551 +0xab
[2022-02-28T16:18:33.349Z] created by gopkg.in/mgo%2ev2.newSocket
[2022-02-28T16:18:33.349Z] 	/go/pkg/mod/gopkg.in/mgo.v2@v2.0.0-20160818020120-3f83fa500528/socket.go:194 +0x1ef
[2022-02-28T16:18:33.349Z]
[2022-02-28T16:18:33.349Z] DONE 10 tests, 4 skipped, 1 failure in 614.784s
[2022-02-28T16:18:33.349Z] Error: failed modules: mongodb
@cmacknz cmacknz added the flaky-test Unstable or unreliable test cases. label Feb 28, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 28, 2022
@cmacknz cmacknz added the Team:Integrations Label for the Integrations team label Feb 28, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 28, 2022
@cmacknz
Copy link
Member Author

cmacknz commented Feb 28, 2022

Duplicates #29208

@cmacknz cmacknz closed this as completed Feb 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Unstable or unreliable test cases. Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cmacknz @elasticmachine