storage: TestReplicateQueueDownReplicate timed out under stress

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/96aafe6226579176f496dfadae78b52d687c3faa

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1009150&tab=buildLog

Slow failing tests:
TestAmbiguousResultErrorOnRetry/non-txn-put - 0.00s

Slow passing tests:
TestAllocatorFullDisks - 11.05s
TestReplicaCommandQueueCancellation - 5.26s
TestGCQueueChunkRequests - 4.45s
TestReplicaCommandQueueCancellationCascade - 2.94s
TestReplicaCommandQueueCancellationRandom - 2.75s
TestGCQueueMakeGCScoreInvariantQuick - 2.63s
TestReplicaCommandQueueCancellationLocal - 1.52s
TestProactiveRaftLogTruncate - 1.39s
TestRaftSSTableSideloadingProposal - 1.32s
TestReplicaCommandQueue - 0.68s

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/b775ca0318125c996305e4ab560e75d2d3471547

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1037228&tab=buildLog

Slow failing tests:
TestReplicateQueueDownReplicate - 0.26s

Slow passing tests:
TestMergeQueue - 105.00s
TestReplicateQueueRebalance - 72.65s
TestRemovePlaceholderRace - 16.94s
TestStoreRangeSplitBackpressureWrites - 14.36s
TestReplicateQueueUpReplicate - 13.05s
TestSplitTriggerRaftSnapshotRace - 12.18s
TestGossipHandlesReplacedNode - 11.24s
TestAllocatorFullDisks - 10.91s
TestWedgedReplicaDetection - 10.87s
TestSnapshotRaftLogLimit - 9.09s
TestStoreSplitFailsAfterMaxRetries - 7.62s
TestReplicaCommandQueueCancellation - 5.21s
TestConsistencyQueueRecomputeStats - 4.37s
TestGCQueueChunkRequests - 4.25s
TestStoreRangeMergeRaftSnapshot - 4.17s
TestNodeLivenessStatusMap - 4.06s
TestGossipFirstRange - 3.88s
TestLogSplits - 3.72s
TestStoreRangeMergeUninitializedLHSFollower - 3.45s
TestStoreSplitOnRemovedReplica - 3.34s

[tbg]

Basically spent close to half an hour like this:

goroutine 122477 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc435eea8d8, 0xc435eea898)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:120 +0x200
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication(0xc420f41ab0, 0x3ea8720, 0xc424878ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:552 +0x2f0
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster(0x3f0e4e0, 0xc43657c780, 0x5, 0x3ea6260, 0xc420796ee0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:172 +0xa03
github.com/cockroachdb/cockroach/pkg/storage_test.TestReplicateQueueDownReplicate(0xc43657c780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replicate_queue_test.go:195 +0x1a4
testing.tRunner(0xc43657c780, 0x37201a0)
	/usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x565

Perhaps something is getting starved by the scanner timings?

	tc := testcluster.StartTestCluster(t, replicaCount+2,
		base.TestClusterArgs{
			ReplicationMode: base.ReplicationAuto,
			ServerArgs: base.TestServerArgs{
				ScanMinIdleTime: time.Millisecond,
				ScanMaxIdleTime: time.Millisecond,
				Knobs: base.TestingKnobs{
					Store: &storage.StoreTestingKnobs{
						// Prevent the merge queue from immediately discarding our splits.
						DisableMergeQueue: true,
					},
				},
			},
		},
	)