Comprehensive Limiters for IO and CPU

[flowbehappy]

Currently, we only have IO limiters for background tasks. Which is not enough. For example, a heavy read task can consume lots of IO, and slow down the write processes, eventually cause the write stall or high Wait Index Duration.

Ideally, we should have limiters, both IO and CPU, for

• Read threads
• Frontend write threads
• Background write threads

And the threshold is better to be self-adjusted. Since CPU is known to be very complicated to limit, we can start with IO limiters.

[JinheLin]

Design doc: https://docs.google.com/document/d/16wEKJHRwfQ8YBXsYIu-z2WmTkBYQZiMP-piuqUomAyg/edit#heading=h.fgvlsvd818w3

[JinheLin]

Subtask(Plan)

• Implement configurable I/O rate limit for background and foreground write request. (doing: deving and self-testing)
• Use epbf to monitoring the I/O statistics, such as read/write bandwith and latency.
• Maybe use Block IO Controller to limit the total read I/O bandwith.
• Try to implement the auto-tune IO rate limiter.

[JaySon-Huang]

Update docs in

• TiFlash io rate limit configuration. docs-cn#6820
• TiFlash configuration: update io_rate_limit docs#6176