roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness failed

[cockroach-teamcity]

roachtest.follower-reads/survival=region/locality=regional/reads=bounded-staleness failed with artifacts on release-21.2 @ f51352920c0af2bbede32dab8fee6424fc27a911:

The test failed on branch=release-21.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=region/locality=regional/reads=bounded-staleness/run_1
	follower_reads.go:545,follower_reads.go:327,follower_reads.go:57,test_runner.go:777: 11 latency values ([20.05401ms 20.119546ms 20.119546ms 20.185082ms 20.05401ms 20.119546ms 20.05401ms 20.119546ms 20.250618ms 20.250618ms 20.250618ms]) are above target latency 20ms, 4 permitted

Reproduce

See: roachtest README

Same failure on other branches

• roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed #71244 roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed [C-test-failure O-roachtest O-robot branch-master skipped-test]
• roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness failed #70011 roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness failed [C-test-failure O-roachtest O-robot T-kv branch-master skipped-test]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.follower-reads/survival=region/locality=regional/reads=bounded-staleness failed with artifacts on release-21.2 @ 33ec37ae94ff96d2f5804506810fb436b186bc32:

The test failed on branch=release-21.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=region/locality=regional/reads=bounded-staleness/run_1
	follower_reads.go:545,follower_reads.go:327,follower_reads.go:57,test_runner.go:777: 11 latency values ([20.38169ms 20.119546ms 20.447226ms 21.03705ms 20.840442ms 20.316154ms 20.05401ms 20.512762ms 20.643834ms 20.119546ms 20.512762ms]) are above target latency 20ms, 4 permitted

Reproduce

See: roachtest README

Same failure on other branches

• roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed #71244 roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness/insufficient-quorum failed [C-test-failure O-roachtest O-robot branch-master skipped-test]
• roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness failed #70011 roachtest: follower-reads/survival=region/locality=regional/reads=bounded-staleness failed [C-test-failure O-roachtest O-robot T-kv branch-master skipped-test]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

[tbg]

@nvanbenschoten could you determine whether to remove release-blocker here?

[nvanbenschoten]

As we've seen in the past, these 20ms latencies are still follower reads, they're just slower than the arbitrary limit we set to detect follower reads. @aayushshah15 noticed in #73600 that we are running this test on underpowered machines, which explains why we occasionally see spikes in tail latency.

@aayushshah15 do you mind backporting that PR to release-21.2? I think that's enough to close this issue.

[nvanbenschoten]

Closing thanks to #74424.