kv/kvserver: TestReplicateQueueDeadNonVoters and TestReplicateQueueSwapVotersWithNonVoters timing out

[sumeerbhola]

Under race
https://teamcity.cockroachdb.com/viewLog.html?buildId=3036495&tab=buildResultsDiv&buildTypeId=Cockroach_UnitTests_Testrace

https://teamcity.cockroachdb.com/viewLog.html?buildId=3036336&tab=buildResultsDiv&buildTypeId=Cockroach_UnitTests_Testrace

https://teamcity.cockroachdb.com//viewLog.html?buildId=3034938&buildTypeId=Cockroach_MergedExtendedCi

[aayushshah15]

When I attempt to reproduce these failures under high concurrency, these tests simply do not make progress due to either node liveness getting wedged or gossip getting wedged. Under low concurrency (i.e. with -p 2 or -p 4 on beefy roachprod machines), I cannot get any failures/timeouts whatsoever.

Funnily enough, almost all the other TestReplicateQueue... tests in that file are already skipped for the same reason and the one that isn't (TestReplicateQueueDecommissioningNonVoters) is failing the exact same way like the two tests this issue is about.

These seem to be another instance of spurious testrace timeouts that are a result of CI machines being too pegged. @tbg @erikgrinaker I'd like to understand how y'all feel about declaring bankruptcy here again. It's unsatisfying, but I'm also not sure there's much else to be done.

[ajwerner]

What happens if you make raft timeouts effectively not happen by setting the timeout to something very large? I often wonder whether setting these timeouts to be extremely large out to be the norm under stress/stressrace.

--- a/pkg/kv/kvserver/replicate_queue_test.go
+++ b/pkg/kv/kvserver/replicate_queue_test.go
@@ -522,6 +522,9 @@ func TestReplicateQueueDeadNonVoters(t *testing.T) {
                        base.TestClusterArgs{
                                ReplicationMode: base.ReplicationAuto,
                                ServerArgs: base.TestServerArgs{
+                                       RaftConfig: base.RaftConfig{
+                                               RaftElectionTimeoutTicks: 1000,
+                                       },
                                        Knobs: base.TestingKnobs{
                                                NodeLiveness: kvserver.NodeLivenessTestingKnobs{
                                                        StorePoolNodeLivenessFn: func(

[aayushshah15]

That's a good point.

I tried with RaftElectionTimeoutTicks: 1000 but kept running into a bunch of other slowness that impedes the TestCluster setup. For instance, with ReplicationAuto, the test cluster will try to upreplicate before returning from its Start() method. However, it's hard to make this succeed, as the replicate queue keeps butting its head against a bunch of our context cancellation deadlines.

kv/kvserver/queue.go:1093  [n1,raftsnapshot,s1,r4/1:/System{/tsd-tse}] 2809  operation "raftsnapshot queue process replica 4" timed out after 1m0s: context deadline exceeded
...
[n2,replicate,s2,r19/2:/Table/2{3-4}] 2894  failed to rollback LEARNER n3,s3, abandoning it for the replicate queue: operation "learner rollback" timed out after 10s: change replicas of r19 failed: log-range-event: context deadline exceeded

We have a ton of these tests that create some scenario and drain the {replicate,merge,split}Queue to assert that the queue actually took some action, and its hard to see how we could make these tests non-flakey if they're always going to be running on overloaded CI machines.

[erikgrinaker]

These seem to be another instance of spurious testrace timeouts that are a result of CI machines being too pegged. @tbg @erikgrinaker I'd like to understand how y'all feel about declaring bankruptcy here again. It's unsatisfying, but I'm also not sure there's much else to be done.

Running timing-sensitive tests with >= 3 nodes under race has never worked well for me, and tends to create far more noise than signal. Skipping them under race seems fine to me.