roachtest: perturbation/metamorphic/restart failed

[cockroach-teamcity]

roachtest.perturbation/metamorphic/restart failed with artifacts on master @ b3fec61ca90095c664f6432af864f18e9946f8bb:

(soon.go:60).SucceedsWithin: condition failed to evaluate within 10m0s: from validation_check.go:161: tokens not returned for kvflowcontrol.tokens.send.elastic: deducted 3712570 returned 2970056
test artifacts and logs in: /artifacts/perturbation/metamorphic/restart/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=16
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false
• ssd=1

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-45121

[kvoli]

I think w/ the metamorphic variation, we may need to extend the timeout further.

[arulajmani]

Metamorphic perturbation test; removing the release blocker label, adding C-bug, A-testing and assigning to @andrewbaptist.

[andrewbaptist]

Since this is a non-default configuration which only occurs with the diskBandwidthLimiter, I'm going to leave this open for now rather than extend the timer past 10m and lower to a P3. Without the limiter, the tokens are returned immediately, however the impact of the perturbation is much higher.

@aadityasondhi this might be of interest to you as well as it is fairly easy to reproduce. The "default" run configuration with the limiter takes multiple minutes to return the tokens, so I think there is some issue with the way we are computing the admitted rate.