executor: add an OOMAction for copIterator to adaptive control the memory usage #19246

Yisaer · 2020-08-17T11:28:37Z

What problem does this PR solve?

Currently TableReader will consume too much memory and have no oom action.

Issue Number: close #xxx
ref #16104

What is changed and how it works?

Add one oom action for the coprocessor in non keep-order case. As copIterator in non keepOrder case have already supported sendRate. The oom action strategy is to reduce the sendrate ticket and drain out copResponse in response channel while the copWroker have been suspended when the oom action have been triggered.

If the ticket for the sendrate have been reduced to 1 , the oom action will delegate to the fallback action.

Here is the memory status for backuping 200G data with v4.0.2 PD and TIKV

logs during backup

[root@172.16.4.4 gaosong]# kubectl logs -c tidb -n gaosong basic-tidb-0 -f | grep taskRateLimit
[2020/08/21 05:06:31.047 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1179570309] [quota=1073741824] [maxConsumed=1179570309] [tearedTicket=0] [ticketTotal=15]
[2020/08/21 05:06:47.791 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:06:49.720 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083889923] [quota=1073741824] [maxConsumed=2480231489] [tearedTicket=1] [ticketTotal=15]
[2020/08/21 05:07:06.971 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:07:08.933 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083926344] [quota=1073741824] [maxConsumed=2480231489] [tearedTicket=2] [ticketTotal=15]
[2020/08/21 05:07:27.230 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:07:29.378 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083885830] [quota=1073741824] [maxConsumed=2480231489] [tearedTicket=3] [ticketTotal=15]
[2020/08/21 05:07:48.401 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:07:50.292 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083942369] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=4] [ticketTotal=15]
[2020/08/21 05:08:08.136 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:08:10.177 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083918969] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=5] [ticketTotal=15]
[2020/08/21 05:08:27.572 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:08:29.945 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083918330] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=6] [ticketTotal=15]
[2020/08/21 05:08:48.998 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:08:51.700 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083917546] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=7] [ticketTotal=15]
[2020/08/21 05:09:11.398 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:09:14.120 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083855143] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=8] [ticketTotal=15]
[2020/08/21 05:09:31.630 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:09:34.553 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083889738] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=9] [ticketTotal=15]
[2020/08/21 05:09:54.937 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:09:57.860 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083885191] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=10] [ticketTotal=15]
[2020/08/21 05:10:15.279 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:10:19.209 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083882653] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=11] [ticketTotal=15]
[2020/08/21 05:10:38.468 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:10:43.895 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083872351] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=12] [ticketTotal=15]
[2020/08/21 05:11:04.014 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]
[2020/08/21 05:11:14.353 +00:00] [INFO] [coprocessor.go:1275] ["memory exceeds quota, mark taskRateLimitAction exceed signal."] [consumed=1083853584] [quota=1073741824] [maxConsumed=2492900197] [tearedTicket=13] [ticketTotal=15]
[2020/08/21 05:11:34.720 +00:00] [INFO] [coprocessor.go:693] ["taskRateLimitAction Broadcast"]

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test

Release note

No release note

zz-jason · 2020-08-24T03:03:26Z

@Yisaer can we reduce the log size?

Yisaer · 2020-08-24T10:35:11Z

@Yisaer can we reduce the log size?

updated, the unnecessary log param is removed now.

store/tikv/coprocessor.go

Yisaer · 2020-09-01T05:04:49Z

/run-unit-test

XuHuaiyu · 2020-09-21T11:51:28Z

LGTM

XuHuaiyu · 2020-09-21T11:52:46Z

/run-all-tests

XuHuaiyu · 2020-09-21T11:58:26Z

/run-all-tests

wshwsh12

Almost LGTM

store/tikv/coprocessor.go

wshwsh12 · 2020-09-22T06:01:03Z

LGTM

wshwsh12 · 2020-09-22T06:01:17Z

/merge

ti-srebot · 2020-09-22T06:03:38Z

/run-all-tests

ti-srebot · 2020-09-22T06:05:43Z

@Yisaer merge failed.

XuHuaiyu · 2020-09-22T06:15:38Z

/run-tics-test
/run-unit-test

zz-jason · 2020-09-27T08:54:12Z

@Yisaer Could you prepare a detailed test report for this PR? We may need to consider these parts:

What's the performance influence for normal queries?
What's the average speed for dump different size data, for example, 100 GB, 200 GB, 500 GB, 1 TB?
What's the average time it takes to shrink memory usage to satisfy the memory quota for a query?
When will this optimization fail to prevent OOM? Would it fail if the system available memory and the memory quota for a query are both 1 GB? If it is, what's the expected memory quota configuration for a specific system memory?

Yisaer · 2020-10-09T09:08:04Z

@Yisaer Could you prepare a detailed test report for this PR? We may need to consider these parts:

What's the performance influence for normal queries?

What's the average speed for dump different size data, for example, 100 GB, 200 GB, 500 GB, 1 TB?

What's the average time it takes to shrink memory usage to satisfy the memory quota for a query?

When will this optimization fail to prevent OOM? Would it fail if the system available memory and the memory quota for a query are both 1 GB? If it is, what's the expected memory quota configuration for a specific system memory?

got it.

…mory usage (pingcap#19246)

Yisaer added 2 commits August 17, 2020 18:43

add sendra control

9537d05

add oom

2cfb345

Yisaer requested a review from a team as a code owner August 17, 2020 11:28

Yisaer requested review from lzmhhh123 and removed request for a team August 17, 2020 11:28

Yisaer marked this pull request as draft August 17, 2020 11:28

github-actions bot added the sig/executor label Aug 17, 2020

AndreMouche removed the sig/executor label Aug 20, 2020

github-actions bot added the sig/execution SIG execution label Aug 20, 2020

Yisaer added 2 commits August 20, 2020 19:59

Merge remote-tracking branch 'upstream/master' into add_send_rate_2_oom

dad18db

Update table_reader.go

0ce5275

Yisaer changed the title ~~[DNM] executor: Add coprocessor oom action~~ executor: Add coprocessor oom action Aug 21, 2020

Yisaer marked this pull request as ready for review August 21, 2020 06:16

remove useless panic

d6be96e

XuHuaiyu requested review from fzhedu and wshwsh12 August 24, 2020 02:51

Yisaer added 2 commits August 24, 2020 12:21

Merge remote-tracking branch 'upstream/master' into add_send_rate_2_oom

3b2c0c6

remove unnecessary log

1182cea

use debug

61d491f

wshwsh12 reviewed Aug 28, 2020

View reviewed changes

store/tikv/coprocessor.go Show resolved Hide resolved

store/tikv/coprocessor.go Outdated Show resolved Hide resolved

address the comments

ab38994

Yisaer requested a review from wshwsh12 August 31, 2020 04:14

address the comment

87935da

Yisaer added 4 commits September 1, 2020 14:03

add unit test

6ae13f5

add unit test

0c2e555

address the comment

94fc5a1

Merge remote-tracking branch 'upstream/master' into add_send_rate_2_oom

0179bae

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 21, 2020

Merge branch 'master' into add_send_rate_2_oom

63e021c

fix race

6b69a82

wshwsh12 reviewed Sep 21, 2020

View reviewed changes

store/tikv/coprocessor.go Outdated Show resolved Hide resolved

store/tikv/coprocessor.go Outdated Show resolved Hide resolved

store/tikv/coprocessor.go Outdated Show resolved Hide resolved

Yisaer added 2 commits September 22, 2020 11:44

address the comment

5c5bd9e

address the comment

3d8478d

wshwsh12 reviewed Sep 22, 2020

View reviewed changes

store/tikv/coprocessor.go Outdated Show resolved Hide resolved

address the comment

e96164e

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Sep 22, 2020

ti-srebot approved these changes Sep 22, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Sep 22, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2020

Merge branch 'master' into add_send_rate_2_oom

198f8bd

wshwsh12 merged commit 205f021 into pingcap:master Sep 22, 2020

XuHuaiyu mentioned this pull request Oct 20, 2020

change the default value of oom-action from log to cancel #20527

Closed

3 tasks

Yisaer added a commit to Yisaer/tidb that referenced this pull request Nov 25, 2020

executor: add an OOMAction for copIterator to adaptive control the me…

262617c

…mory usage (pingcap#19246)

Yisaer mentioned this pull request Nov 25, 2020

store, executor: add an OOMAction for copIterator to adaptive control the memory usage #21297

Merged

XuHuaiyu mentioned this pull request Sep 15, 2022

set the default value of tidb_enable_rate_limit_action to false cause it'll block mem_quota_query takeing effect #37810

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: add an OOMAction for copIterator to adaptive control the memory usage #19246

executor: add an OOMAction for copIterator to adaptive control the memory usage #19246

Yisaer commented Aug 17, 2020 •

edited by ti-srebot

Loading

zz-jason commented Aug 24, 2020

Yisaer commented Aug 24, 2020

Yisaer commented Sep 1, 2020

XuHuaiyu commented Sep 21, 2020

XuHuaiyu commented Sep 21, 2020

XuHuaiyu commented Sep 21, 2020

wshwsh12 left a comment

wshwsh12 commented Sep 22, 2020

wshwsh12 commented Sep 22, 2020

ti-srebot commented Sep 22, 2020

ti-srebot commented Sep 22, 2020

XuHuaiyu commented Sep 22, 2020

zz-jason commented Sep 27, 2020

Yisaer commented Oct 9, 2020

executor: add an OOMAction for copIterator to adaptive control the memory usage #19246

executor: add an OOMAction for copIterator to adaptive control the memory usage #19246

Conversation

Yisaer commented Aug 17, 2020 • edited by ti-srebot Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

zz-jason commented Aug 24, 2020

Yisaer commented Aug 24, 2020

Yisaer commented Sep 1, 2020

XuHuaiyu commented Sep 21, 2020

XuHuaiyu commented Sep 21, 2020

XuHuaiyu commented Sep 21, 2020

wshwsh12 left a comment

Choose a reason for hiding this comment

wshwsh12 commented Sep 22, 2020

wshwsh12 commented Sep 22, 2020

ti-srebot commented Sep 22, 2020

ti-srebot commented Sep 22, 2020

XuHuaiyu commented Sep 22, 2020

zz-jason commented Sep 27, 2020

Yisaer commented Oct 9, 2020

Yisaer commented Aug 17, 2020 •

edited by ti-srebot

Loading