Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestRandomConcurrentAdminChangeReplicasRequests failed #55913

Closed
cockroach-teamcity opened this issue Oct 23, 2020 · 8 comments · Fixed by #55937
Closed

kv/kvserver: TestRandomConcurrentAdminChangeReplicasRequests failed #55913

cockroach-teamcity opened this issue Oct 23, 2020 · 8 comments · Fixed by #55937
Assignees
@ajwerner
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 207 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 351 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	439.207s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 23, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 23, 2020
@rytaft
Copy link
Collaborator

rytaft commented Oct 23, 2020

This is the same data race as the recent failure in #50032.

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 1166 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 474 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	411.195s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@33e3abc57cccedd86dd1b2e1d58feb075fcb8dd4:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 1539 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 2386 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	485.271s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 2662 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 1957 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	390.236s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 1496 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 79 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomConcurrentAdminChangeReplicasRequests462284672
--- FAIL: TestRandomConcurrentAdminChangeReplicasRequests (7.65s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestRandomConcurrentAdminChangeReplicasRequests462284672
    test_log_scope.go:63: use -show-logs to present logs inline
    testing.go:853: race detected during execution of test
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 547 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 1264 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	415.600s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:157 +0x1e5
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:139 +0x273
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 1872 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 1823 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	418.561s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestRandomConcurrentAdminChangeReplicasRequests failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:78 +0x423
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x108
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1874 +0xa5a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1495 +0x3cd
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1133 +0x1eca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:772 +0xab3
  github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:219 +0xfc
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:744 +0x1ad
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).send-fm()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:726 +0xf6
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:654 +0x182
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:677 +0xd1
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminChangeReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:592 +0x166
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2230 +0x1e6
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x38

Goroutine 2369 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:128 +0x100
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199

Goroutine 1615 (running) created at:
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2237 +0x6e2
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:128 +0x100
  github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestRandomConcurrentAdminChangeReplicasRequests()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_replica_test.go:2200 +0x190
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
==================
FAIL	github.com/cockroachdb/cockroach/pkg/kv/kvserver	432.980s
More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestRandomConcurrentAdminChangeReplicasRequests PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@rytaft rytaft mentioned this issue Oct 24, 2020
Merged
@craig craig bot closed this as completed in #55937 Oct 24, 2020
@craig craig bot closed this as completed in 7fa301f Oct 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

tracing: avoid data races w.r.t noopSpan tbg/cockroach
3 participants
@ajwerner @rytaft @cockroach-teamcity