Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IORateLimiter support auto-tune. #2594

Merged
merged 7 commits into from
Aug 5, 2021
Merged

IORateLimiter support auto-tune. #2594

merged 7 commits into from
Aug 5, 2021

Conversation

JinheLin
Copy link
Contributor

@JinheLin JinheLin commented Aug 5, 2021 •
edited
Loading

What problem does this PR solve?

Problem Summary:

This PR is a sub-task of issue: #1720. It implements the auto-tune ability between limiters. You can refer the design document (Chinese) for more detail.

What is changed and how it works?

Proposal: TiFlash IO Rate Limiter

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch:

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

Release note

  • Make I/O rate limiter more comprehensive.

Testing example

68747470733a2f2f6c68332e676f6f676c6575736572636f6e74656e742e636f6d2f3146697a54786d314578774b7a68736a6a4a6e6865544b314b4236543850726777376b436368677a734f626542366d596b37385a524a464a46692d4b545864517a4c394a484762

  1. Set foreground read limit is 5MB/s and max_bytes_per_sec is 100MB/s.
  2. About 17:59 turn on auto-tune config，and the foreground read limit is automatically adjusted to 96MB（from log）.
  3. The foreground read limiter's max_bytes_per_sec change sequence: 5242880 -> 10485760 -> 13107200 -> 52254038 -> 77070328 -> 90592580 -> 97065213 -> 99916307 -> 100649118 -> 100663248.

New Metrics

image

grafana metric plane example: http://172.16.5.81:19200/d/SVbh2xUWk/jinhe-test-bench-tiflash-summary?viewPanel=86&orgId=1&from=1627880155161&to=1627890502256

@JinheLin @flowbehappy
Update dbms/src/Server/StorageConfigParser.h
39067e1 
Co-authored-by: Flowyi <flowbehappy@gmail.com>
@flowbehappy flowbehappy self-requested a review August 5, 2021 12:16
flowbehappy
flowbehappy approved these changes Aug 5, 2021
View reviewed changes
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 5, 2021
@JinheLin
Copy link
Contributor Author

JinheLin commented Aug 5, 2021

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 5, 2021
@ti-srebot
Copy link
Collaborator

Your auto merge job has been accepted, waiting for:

  • 2592
  • 2533

@ti-srebot
Copy link
Collaborator

/run-all-tests

@ti-srebot
Copy link
Collaborator

@JinheLin merge failed.

@JinheLin
Copy link
Contributor Author

JinheLin commented Aug 5, 2021

/merge

@ti-srebot
Copy link
Collaborator

/run-all-tests

@ti-srebot ti-srebot merged commit e3c6efa into pingcap:master Aug 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

Assignees
No one assigned
Labels
status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JinheLin @ti-srebot @flowbehappy