kvflowcontrol: token return can take a long time with diskBandwidthLimit set

[andrewbaptist]

Describe the problem

A cluster with diskBandwidthLimit enabled can take 10s of minutes to return elastic tokens on an idle system.

To Reproduce

Run many of the perturbation tests with a disk bandwidth set to 350MiB and notice that the token return can take 10+ minutes.

e.g.

PERTURBATION_OVERRIDE=acMode=diskBandwidthLimit=350MiB roachtest run perturbation/dev/addNode -l

Expected behavior
During the end period of the test there is no longer an IO and the CPU and disks are sitting idle (<5% utilization). It is expected that the tokens would be returned sooner.

Its not clear if the problem is with the disk bandwidth limit or the RACv2 handling of it.

This issue is intented to track the workaround to wait longer for token returns when this setting is enabled.

Jira issue: CRDB-45358

[blathers-crl]

Hi @andrewbaptist, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[andrewbaptist]

For the latest run n12 was restarted at 15:51:47 and all SQL traffic stopped at 15:56:51.

Here is a graph of a test run where it sits for 10m without returning tokens:

Logs:
cockroach.1.log
cockroach.12.log