Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tbot race condition flakiness #15843

Closed
ibeckermayer opened this issue Aug 25, 2022 · 3 comments
Closed

tbot race condition flakiness #15843

ibeckermayer opened this issue Aug 25, 2022 · 3 comments
Assignees
@timothyb89 @strideynet
Labels
flaky tests machine-id

Comments

@ibeckermayer
Copy link
Contributor

ibeckermayer commented Aug 25, 2022 •
edited
Loading

Failure

Link(s) to logs

Relevant snippet

It looks like everything in renew_test.go and tbot_test.go caused a race condition to be detected, along the lines of:

testing.go:1312: race detected during execution of test
@ibeckermayer ibeckermayer mentioned this issue Aug 25, 2022
Closed
@strideynet
Copy link
Contributor

Thanks for raising this.

@strideynet
Copy link
Contributor

strideynet commented Aug 25, 2022 •
edited
Loading

Race detector has managed to give us the read points, but hasn't been able to recover where the write occurred, which makes it a tad harder to determine which of the tests this is occurring within.

==================
WARNING: DATA RACE
Read at 0x00c000f00450 by goroutine 545:
  reflect.maplen()
      /opt/go/src/runtime/map.go:1392 +0x0
  reflect.Value.Len()
      /opt/go/src/reflect/value.go:1575 +0x268
  github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func31()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:598 +0x96
  github.com/gogo/protobuf/proto.(*mergeInfo).merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:139 +0x690
  github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func28()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:555 +0x44
  github.com/gogo/protobuf/proto.(*mergeInfo).merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:139 +0x690
  github.com/gogo/protobuf/proto.(*InternalMessageInfo).Merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:50 +0x18f
  github.com/gravitational/teleport/api/types.(*AppV3).XXX_Merge()
      /workspace/api/types/types.pb.go:1893 +0x4e
  github.com/gogo/protobuf/proto.Merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/clone.go:95 +0x6a7
  github.com/gogo/protobuf/proto.Clone()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/clone.go:52 +0x31a
  github.com/gravitational/teleport/api/types.(*AppV3).Copy()
      /workspace/api/types/app.go:277 +0x35
  github.com/gravitational/teleport/lib/srv/app.(*Server).getServerInfo()
      /workspace/lib/srv/app/server.go:410 +0x7e
  github.com/gravitational/teleport/lib/srv/app.(*Server).getServerInfoFunc.func1()
      /workspace/lib/srv/app/server.go:401 +0x4c
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).fetch()
      /workspace/lib/srv/heartbeat.go:333 +0x5e
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).fetchAndAnnounce()
      /workspace/lib/srv/heartbeat.go:602 +0x30
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run()
      /workspace/lib/srv/heartbeat.go:276 +0xc5
  github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat.func1()
      /workspace/lib/srv/app/server.go:378 +0x39

Previous write at 0x00c000f00450 by goroutine 55:
  [failed to restore the stack]

Goroutine 545 (running) created at:
  github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat()
      /workspace/lib/srv/app/server.go:378 +0x2d1
  github.com/gravitational/teleport/lib/srv/app.(*Server).startApp()
      /workspace/lib/srv/app/server.go:290 +0xa4
  github.com/gravitational/teleport/lib/srv/app.(*Server).registerApp()
      /workspace/lib/srv/app/server.go:453 +0x9a
  github.com/gravitational/teleport/lib/srv/app.(*Server).Start()
      /workspace/lib/srv/app/server.go:503 +0x214
  github.com/gravitational/teleport/lib/service.(*TeleportProcess).initApps.func1()
      /workspace/lib/service/service.go:4371 +0x1884
  github.com/gravitational/teleport/lib/service.(*LocalService).Serve()
      /workspace/lib/service/supervisor.go:539 +0x3d
  github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
      /workspace/lib/service/supervisor.go:276 +0x1fd

Goroutine 55 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /opt/go/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /opt/go/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /opt/go/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /opt/go/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:111 +0x3a9
==================
==================
WARNING: DATA RACE
Read at 0x00c000296638 by goroutine 545:
  reflect.typedmemmove()
      /opt/go/src/runtime/mbarrier.go:178 +0x0
  reflect.copyVal()
      /opt/go/src/reflect/value.go:1834 +0x6e
  reflect.Value.MapIndex()
      /opt/go/src/reflect/value.go:1624 +0x448
  github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func31()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:621 +0x5d8
  github.com/gogo/protobuf/proto.(*mergeInfo).merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:139 +0x690
  github.com/gogo/protobuf/proto.(*mergeInfo).computeMergeInfo.func28()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:555 +0x44
  github.com/gogo/protobuf/proto.(*mergeInfo).merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:139 +0x690
  github.com/gogo/protobuf/proto.(*InternalMessageInfo).Merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/table_merge.go:50 +0x18f
  github.com/gravitational/teleport/api/types.(*AppV3).XXX_Merge()
      /workspace/api/types/types.pb.go:1893 +0x4e
  github.com/gogo/protobuf/proto.Merge()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/clone.go:95 +0x6a7
  github.com/gogo/protobuf/proto.Clone()
      /go/pkg/mod/github.com/gravitational/protobuf@v1.3.2-0.20201123192827-2b9fcfaffcbf/proto/clone.go:52 +0x31a
  github.com/gravitational/teleport/api/types.(*AppV3).Copy()
      /workspace/api/types/app.go:277 +0x35
  github.com/gravitational/teleport/lib/srv/app.(*Server).getServerInfo()
      /workspace/lib/srv/app/server.go:410 +0x7e
  github.com/gravitational/teleport/lib/srv/app.(*Server).getServerInfoFunc.func1()
      /workspace/lib/srv/app/server.go:401 +0x4c
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).fetch()
      /workspace/lib/srv/heartbeat.go:333 +0x5e
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).fetchAndAnnounce()
      /workspace/lib/srv/heartbeat.go:602 +0x30
  github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run()
      /workspace/lib/srv/heartbeat.go:276 +0xc5
  github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat.func1()
      /workspace/lib/srv/app/server.go:378 +0x39

Previous write at 0x00c000296638 by goroutine 55:
  [failed to restore the stack]

Goroutine 545 (running) created at:
  github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat()
      /workspace/lib/srv/app/server.go:378 +0x2d1
  github.com/gravitational/teleport/lib/srv/app.(*Server).startApp()
      /workspace/lib/srv/app/server.go:290 +0xa4
  github.com/gravitational/teleport/lib/srv/app.(*Server).registerApp()
      /workspace/lib/srv/app/server.go:453 +0x9a
  github.com/gravitational/teleport/lib/srv/app.(*Server).Start()
      /workspace/lib/srv/app/server.go:503 +0x214
  github.com/gravitational/teleport/lib/service.(*TeleportProcess).initApps.func1()
      /workspace/lib/service/service.go:4371 +0x1884
  github.com/gravitational/teleport/lib/service.(*LocalService).Serve()
      /workspace/lib/service/supervisor.go:539 +0x3d
  github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
      /workspace/lib/service/supervisor.go:276 +0x1fd

Goroutine 55 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /opt/go/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /opt/go/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /opt/go/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /opt/go/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:111 +0x3a9
==================

@strideynet
Copy link
Contributor

Closing this as stale - please reopen if this recurs but I've not seen this in a while so I'm going to assume various test refactors have taken care of this.

@strideynet strideynet closed this as not planned Won't fix, can't repro, duplicate, stale Jul 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timothyb89 timothyb89

@strideynet strideynet

Labels
flaky tests machine-id
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@timothyb89 @ibeckermayer @strideynet