Improve memtrack #5610
Improve memtrack #5610
Conversation
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
release-note-none
|
This pull request has been accepted and is ready to merge. Commit hash: da1eb24
|
ti-chi-bot
added
the
status/can-merge
|
/rebuild |
|
/run-all-tests |
|
@bestwoody: Your PR was out of date, I have automatically updated it for you. At the same time I will also trigger all tests for you: /run-all-tests If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
ti-chi-bot
removed
the
status/can-merge
|
/merge |
|
@bestwoody: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger If you have any questions about the PR merge process, please refer to pr process. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
This pull request has been accepted and is ready to merge. Commit hash: 102938c
|
ti-chi-bot
added
the
status/can-merge
ti-chi-bot
removed
the
status/can-merge
|
/run-all-tests |
|
/merge |
|
@bestwoody: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger If you have any questions about the PR merge process, please refer to pr process. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
This pull request has been accepted and is ready to merge. Commit hash: 512b177
|
ti-chi-bot
added
the
status/can-merge
Coverage for changed filesCoverage summaryfull coverage report (for internal network access only) |
What problem does this PR solve?
Issue Number: close #5609
Problem Summary:
What is changed and how it works?
Solve the following problems which may cause memory quota not working, in order to reduce OOM cases.
1.Track some kind of object was not tracked before, such as object generated by Protobuf.
2.Fix a bug in memtracker, which causes memory quota not working.
3.Fix bugs in our MPPTask construct/destruct logic, which cause memtracker can‘t be propagated into MPPTunnel & ExchangeReceiver
4. Use a background thread to collect RSS metric periodicaly and check how much it exceed the user setting limit in mem tracker. And throttle when it's too large, to control those un-trackable memory such as: cache used by Allocator 、stack allocated by threads.
How RSS check works：
Besides a simply condition check RSS > limit + 5GB. A dynamic linear check logic is also defined.
An example for how this logic works: if limit is 30GB, and real_rss is 40GB, since then mem alloc will fail when tracked amount > 25GB(30-(10-5)=25), when memory usage turn down below 25GB, all query tasks can continue work, where there is no need to wait the real_rss metric updated in next collect 100ms period. The larger real_rss over than limit+5GB, the lower ceiling of memory tracked amount where we can continue allocating memory.
Check List
Tests
Side effects
Documentation
Release note