Read I/O rate limit for background and foreground read request.

[JinheLin]

What problem does this PR solve?

Problem Summary:

This PR is a sub-task of issue: #1720. It implements configurable I/O rate limit for background and foreground read request base on the statistic data of /proc/<pid>/io and /proc/<pid>/tasks/<tid>/io. You can refer the design document (Chinese) for more detail.

What is changed and how it works?

Proposal: TiFlash IO Rate Limiter

What's Changed:

• New Configurations:

  storage.io-rate-limit.max-bytes-per-sec: 0
  storage.io-rate-limit.max-read-bytes-per-sec: 0
  storage.io-rate-limit.max-write-bytes-per-sec: 0
  

  max-bytes-per-sec is the maximum bandwidth for I/O limiter.

  max-read-bytes-per-sec and max-write-bytes-per-sec is the maximum bandwidth for read and write I/O limiter.

  Also, you can refer the document (Chinese) for more detail.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch:

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

Release note

• Make I/O rate limiter more comprehensive.

---

Testing results

Limit the foreground read to 1MB/s, 2MB/s, 3MB/s, 60MB/s.Below is the screenshot of the testing results: yellow line is the bandwidth of the disk, red line is the statistic data of the limiter.

Before each test, we need to drop the page cache by the commands:

• sync
• echo 3 > /proc/sys/vm/drop_caches

Limit the foreground read to 1MB/s

Limit the foreground read to 2MB/s

Limit the foreground read to 3MB/s

Limit the foreground read to 60MB/s

Grafana I/O limiter example link: http://172.16.5.59:19169/d/SVbh2xUWk/jinhe-test-bench-tiflash-summary?orgId=1&from=1626766214135&to=1626774127373&viewPanel=84

[JinheLin]

/run-all-tests

[JinheLin]

/run-all-tests

[JinheLin]

/run-all-tests

[flowbehappy]

BTW, why RateLimiter.h is put in Encryption?

[JinheLin]

It was in Encryption before.

@lidezhu Do you know why?

[lidezhu]

Comment it in https://github.com/pingcap/tics/blob/master/dbms/src/Encryption/RateLimiter.h#L11. Put it simply, just to avoid the dependency of the io module on encryption module which makes the compile of some clickhouse unit test fail.

[flowbehappy]

LGTM with a comment

[JinheLin]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@JinheLin merge failed.

[JinheLin]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 2577
• 2579

[ti-srebot]

/run-all-tests

[ti-srebot]

@JinheLin merge failed.

[JinheLin]

/run-all-tests

[JinheLin]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 2573
• 2571
• 2454

[ti-srebot]

/run-all-tests