Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver/kvflowcontrol/rac2: TestRangeController failed #132293

Closed
cockroach-teamcity opened this issue Oct 10, 2024 · 9 comments
Closed

kv/kvserver/kvflowcontrol/rac2: TestRangeController failed #132293

cockroach-teamcity opened this issue Oct 10, 2024 · 9 comments
Assignees
@pav-kv
Labels
A-replication-admission-control-v2 Related to introduction of replication AC v2 A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-noreuse Prevent automatic commenting from CI test failures

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 10, 2024 •
edited by exalate-issue-sync bot
Loading

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed on master @ d063517b3f15b1d0ec43206802acc7e3981e5d5b:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1330 lock 0xc0004f2c88
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1330 lock 0xc0004f2dc8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-42929

Epic CRDB-37515
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team labels Oct 10, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Oct 10, 2024
@github-project-automation github-project-automation bot moved this to Incoming in Storage Oct 10, 2024
@pav-kv pav-kv added A-replication-admission-control-v2 Related to introduction of replication AC v2 and removed A-storage Relating to our storage engine (Pebble) on-disk storage. labels Oct 10, 2024
@pav-kv
Copy link
Collaborator

pav-kv commented Oct 10, 2024

Dup of #132040.

@pav-kv pav-kv added X-duplicate Closed as a duplicate of another issue. A-testing Testing tools and infrastructure and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 10, 2024
@pav-kv pav-kv self-assigned this Oct 10, 2024
@cockroach-teamcity
Copy link
Member Author

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1349 lock 0xc00034a810
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:198 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:197 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1349 lock 0xc0003fa548
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1365 lock 0xc00031e548
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1365 lock 0xc00031e708
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1364 lock 0xc0001b56c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1364 lock 0xc0001b5888
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:210 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1289 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1401 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@kvoli kvoli added GA-blocker branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 labels Oct 14, 2024
@cockroach-teamcity
Copy link
Member Author

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed on master @ a1b013e763abd1454603985996a45663b6e6bcad:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1294 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1398 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1354 lock 0xc000868a88
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:212 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1294 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1398 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

goroutine 1354 lock 0xc000868c48
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:212 rac2.(*testingRCState).sendStreamString ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:211 rac2.(*testingRCState).sendStreamString ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1294 rac2.TestRangeController.func1.1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117 datadriven.RunTest.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337 datadriven.runDirective.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:341 datadriven.runDirective ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:212 datadriven.runDirectiveOrSubTest ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:182 datadriven.runTestInternal ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144 datadriven.RunTestAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:119 datadriven.RunTest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/range_controller_test.go:1398 rac2.TestRangeController.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425 datadriven.Walk.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436 datadriven.WalkAny ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449 datadriven.WalkAny.func1 ???
github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:26 datadriven.subTest.func1 ???

Parameters:

  • attempt=1
  • deadlock=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@pav-kv
Copy link
Collaborator

pav-kv commented Oct 15, 2024

The issue here is test-only.

In handleReadyEntries stack, mutexes are locked in one order:

  1. replicaSendStream.mu
  2. testingRCRange.mu

In testingRCState.sendStreamString, they are locked in a different order:

  1. testingRCRange.mu
  2. replicaSendStream.mu

The ordering of testingRCRange.mu before replicaSendStream.mu is inconsistent with the ordering in RangeController.

@pav-kv pav-kv added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed GA-blocker labels Oct 15, 2024
@pav-kv pav-kv mentioned this issue Oct 15, 2024
Merged
craig bot pushed a commit that referenced this issue Oct 15, 2024
@pav-kv
Merge #132653
c591473 
132653: rac2: fix mutex ordering issues in tests r=kvoli a=pav-kv

The problem is partly described in #132293 (comment), but the same thing applies to the `scheduledMu` mutex. This PR is a minimal fix to get the `--deadlock` CI failures out the way, though this test needs a better refactoring. The purpose/scope of the extra mutexes in the test harness is unclear, and should be revisited and specified. In a deterministic test, there should be no need for extra mutexes - their existence indicates that either the test is non-deterministic, or the mutexes are mostly redundant (and so we should be clear about their purpose).

Fixes #132040, #132293

Co-authored-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
@nicktrav nicktrav added T-kv KV Team and removed T-storage Storage Team labels Oct 15, 2024
@nicktrav nicktrav removed this from Storage Oct 15, 2024
@cockroach-teamcity
Copy link
Member Author

kv/kvserver/kvflowcontrol/rac2.TestRangeController failed with artifacts on release-24.3 @ ab649c44652184290d1177e989330339f3d8c9ec:

            term=1 index=3 pri=NormalPri size=1MiB
        ----
        t1/s1: eval reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
               send reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
        t1/s2: eval reg=+0 B/+16 MiB ela=-3.0 MiB/+8.0 MiB
               send reg=+0 B/+16 MiB ela=+0 B/+8.0 MiB
        t1/s3: eval reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
               send reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9835/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvflowcontrol/rac2/rac2_test_/rac2_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/testdata/range_controller/send_q_watcher:31:
        stream_state [1 args]
        <no input to command>
        ----
        (n1,s1):1: state=replicate closed=false inflight=[1,4) send_queue=[4,4) precise_q_size=+0 B
        eval deducted: reg=+3.0 MiB ela=+0 B
        eval original in send-q: reg=+0 B ela=+0 B
        NormalPri:
          term=1 index=1  tokens=1048576
          term=1 index=2  tokens=1048576
          term=1 index=3  tokens=1048576
        ++++
        (n2,s2):2: state=replicate closed=false inflight=[1,1) send_queue=[1,4) precise_q_size=+3.0 MiB watching-for-tokens
        eval deducted: reg=+0 B ela=+3.0 MiB
        eval original in send-q: reg=+3.0 MiB ela=+0 B
        ++++
        (n3,s3):3: state=replicate closed=false inflight=[1,4) send_queue=[4,4) precise_q_size=+0 B
        eval deducted: reg=+3.0 MiB ela=+0 B
        eval original in send-q: reg=+0 B ela=+0 B
        NormalPri:
          term=1 index=1  tokens=1048576
          term=1 index=2  tokens=1048576
          term=1 index=3  tokens=1048576
        ++++
        MsgApps sent in pull mode:
         to: 3, lowPri: false entries: [1 2 3]
        ++++
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9835/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvflowcontrol/rac2/rac2_test_/rac2_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/kvflowcontrol/rac2/testdata/range_controller/send_q_watcher:59:
         store_id=2 pri=HighPri tokens=512KiB
        output didn't match expected:
        @@ -1,7 +1,7 @@
         t1/s1: eval reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
                send reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
         t1/s2: eval reg=+0 B/+16 MiB ela=-3.0 MiB/+8.0 MiB
        -       send reg=+512 KiB/+16 MiB ela=+0 B/+8.0 MiB
        +       send reg=+512 KiB/+16 MiB ela=+512 KiB/+8.0 MiB
         t1/s3: eval reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
                send reg=-3.0 MiB/+16 MiB ela=-3.0 MiB/+8.0 MiB
         
    --- FAIL: TestRangeController/send_q_watcher (0.09s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@pav-kv pav-kv added the X-noreuse Prevent automatic commenting from CI test failures label Oct 23, 2024
@pav-kv
Copy link
Collaborator

pav-kv commented Oct 23, 2024 •
edited
Loading

The last failure is different from the one that opened this issue. The initial one is fixed. I put a X-noreuse label so that the next failure, if any, makes a new issue. Leaving this open for now.

@pav-kv
Copy link
Collaborator

pav-kv commented Oct 25, 2024

The new failure came up again in #133426. Since the original failure was fixed, closing this issue.

@pav-kv pav-kv closed this as completed Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pav-kv pav-kv

Labels
A-replication-admission-control-v2 Related to introduction of replication AC v2 A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue. X-noreuse Prevent automatic commenting from CI test failures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nicktrav @pav-kv @cockroach-teamcity @kvoli