store: refine the error handling and retry mechanism for stale read #24956

JmPotato · 2021-05-28T09:08:10Z

Signed-off-by: JmPotato ghzpotato@gmail.com

What problem does this PR solve?

Part of #21094 and #23271.

Refine the error handling and retry mechanism for stale read.

What is changed and how it works?

Update the excluded StoreID list during the retry of a stale read request to make sure TiDB will try on different peers.
Add RegionNotInitialized and DataIsNotReady error handling.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Release note

No release note.

JmPotato · 2021-05-31T04:20:52Z

/run-all-tests

zhouqiang-cl · 2021-05-31T06:18:10Z

/rebuild

JmPotato · 2021-06-01T10:05:16Z

@xhebox @nolouch @Yisaer @djshow832 PTAL

store/tikv/region_cache.go

store/tikv/region_request.go

xhebox

Rest LGTM

store/tikv/region_request.go

store/tikv/region_cache.go

store/tikv/region_request.go

store/tikv/region_cache.go

Yisaer · 2021-06-02T05:26:52Z

store/tikv/region_request.go

+ // Stale Read request will retry the leader or next peer on error,
+ // so we will exclude the PeerID of the requested peer every time.
+ // If the new PeerID keeps being the same with the last one, we
+ // should not continue excluding it to make the opts become bigger.
+ if ctx.isStaleRead && ctx.lastPeerID != ctx.Peer.GetId() {
+ opts = append(opts, WithExcludedPeerIDs([]uint64{ctx.Peer.GetId()}))
+ }


The fallback strategy is kind of different from what I think, I think if the tx_scope is local, we shall find other store which located in same dc. If the tx_scope is global, we shall directly send request to leader.

For now, it works like this:

If the tx_scope is local, it will find other stores located in the same DC according to matched labels and try on different peers one by one on the error. If all peers are tried, it will only retry on the leader until fails.

If the tx_scope is global, it will try on different peers one by one on the error. If all peers are tried, it will only retry on the leader until fails.

It seem that we only need to change the behaviour of the global? Once it fails, it will directly request to the leader rather than try all other peers.

I think it's better to send to leader directly when tx_scope is global

Signed-off-by: JmPotato <ghzpotato@gmail.com>

store/tikv/region_request.go

Yisaer

Rest LGTM except faillback strategy

store/tikv/region_request.go

Signed-off-by: JmPotato <ghzpotato@gmail.com>

store/tikv/region_request.go

Signed-off-by: JmPotato <ghzpotato@gmail.com>

xhebox

Rest LGTM

store/tikv/region_cache.go

Signed-off-by: JmPotato <ghzpotato@gmail.com>

JmPotato · 2021-06-03T07:28:03Z

@xhebox @nolouch @Yisaer PTAL.

xhebox

LGTM

youjiali1995

Please add TestingT otherwise tests in store/tikv package won't run.

Yisaer

Rest LGTM

Yisaer · 2021-06-03T09:13:06Z

store/tikv/region_request.go

+ // Stale Read request will retry the leader or next peer on error,
+ // if txnScope is global, we will only retry the leader by using the WithLeaderOnly option,
+ // if txnScope is local, we will retry both other peers and the leader by the incresing seed.
+ if ctx.tryTimes < 1 && req != nil && req.TxnScope == oracle.GlobalTxnScope && req.GetStaleRead() {


why we need tryTimes and tryTimesLimit?

To make WithLeaderOnly option only be appended once to prevent the unnecessary slice memory reallocation. And tryTimesLimit is used by the unit test to mock different retry times.

JmPotato · 2021-06-03T09:16:13Z

Please add TestingT otherwise tests in store/tikv package won't run.

I have added it in store/tikv/tikv_test.go.

ti-chi-bot · 2021-06-03T09:29:29Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Yisaer
xhebox

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Yisaer · 2021-06-03T09:29:42Z

/merge

ti-chi-bot · 2021-06-03T09:29:44Z

This pull request has been accepted and is ready to merge.

Commit hash: c870d80

ti-chi-bot · 2021-06-03T09:38:36Z

@JmPotato: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 28, 2021

JmPotato force-pushed the stale_read_retry branch from f1f439f to 199de7f Compare May 31, 2021 03:55

JmPotato marked this pull request as ready for review May 31, 2021 03:59

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 31, 2021

JmPotato changed the title ~~store: retry the leader or next peer on the Stale Read error~~ store: refine the error handling and retry mechanism for stale read May 31, 2021

Yisaer suggested changes Jun 1, 2021

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

store/tikv/region_cache.go Outdated Show resolved Hide resolved

store/tikv/region_request.go Outdated Show resolved Hide resolved

store/tikv/region_request.go Outdated Show resolved Hide resolved

JmPotato force-pushed the stale_read_retry branch from a1205de to c94f4b1 Compare June 1, 2021 13:05

xhebox reviewed Jun 2, 2021

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

store/tikv/region_cache.go Outdated Show resolved Hide resolved

store/tikv/region_request.go Outdated Show resolved Hide resolved

Yisaer reviewed Jun 2, 2021

View reviewed changes

JmPotato added 8 commits June 2, 2021 13:33

Retry the leader or next peer on the Stale Read error

83b39ea

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Unifying the behavior of StoreSelectorOption

b5ad9ef

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Add RegionNotInitialized error handling

eb9be91

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Not send a read request to any learner peer

c597221

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Address the comments

e0d2e59

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Fix the panic

f881d3c

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Fix the panic

fe99fe4

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Add the stale read retry test

2d7bd46

Signed-off-by: JmPotato <ghzpotato@gmail.com>

JmPotato force-pushed the stale_read_retry branch from bb8737d to 2d7bd46 Compare June 2, 2021 05:33

Address the comments

aa554ca

Signed-off-by: JmPotato <ghzpotato@gmail.com>

JmPotato force-pushed the stale_read_retry branch from b31dcf8 to aa554ca Compare June 2, 2021 06:25

Refine the code

f7d4718

Signed-off-by: JmPotato <ghzpotato@gmail.com>

nolouch reviewed Jun 2, 2021

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

Yisaer reviewed Jun 2, 2021

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

Use seed to try on different peers

884d469

Signed-off-by: JmPotato <ghzpotato@gmail.com>

JmPotato force-pushed the stale_read_retry branch from ea12beb to 884d469 Compare June 2, 2021 16:36

Yisaer reviewed Jun 3, 2021

View reviewed changes

store/tikv/region_request.go Outdated Show resolved Hide resolved

JmPotato requested a review from a team as a code owner June 3, 2021 06:53

JmPotato requested review from wshwsh12 and removed request for a team June 3, 2021 06:53

Refine the test

ce07699

Signed-off-by: JmPotato <ghzpotato@gmail.com>

xhebox reviewed Jun 3, 2021

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

Refine the code

5808577

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Merge branch 'master' into stale_read_retry

a382034

xhebox approved these changes Jun 3, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 3, 2021

Merge branch 'master' into stale_read_retry

c870d80

JmPotato requested review from nolouch and Yisaer June 3, 2021 08:10

github-actions bot added the sig/execution SIG execution label Jun 3, 2021

youjiali1995 reviewed Jun 3, 2021

View reviewed changes

Yisaer reviewed Jun 3, 2021

View reviewed changes

JmPotato mentioned this pull request Jun 3, 2021

raftstore: add the RegionNotInitialized error tikv/tikv#10298

Merged

Yisaer approved these changes Jun 3, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 3, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 3, 2021

ti-chi-bot added 2 commits June 3, 2021 17:29

Merge branch 'master' into stale_read_retry

662dfc9

Merge branch 'master' into stale_read_retry

932d95a

ti-chi-bot merged commit 69274d8 into pingcap:master Jun 3, 2021

JmPotato deleted the stale_read_retry branch June 3, 2021 10:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

store: refine the error handling and retry mechanism for stale read #24956

store: refine the error handling and retry mechanism for stale read #24956

JmPotato commented May 28, 2021 •

edited

Loading

JmPotato commented May 31, 2021

zhouqiang-cl commented May 31, 2021

JmPotato commented Jun 1, 2021

xhebox left a comment

Yisaer Jun 2, 2021

JmPotato Jun 2, 2021

Yisaer Jun 2, 2021

Yisaer left a comment

xhebox left a comment

JmPotato commented Jun 3, 2021

xhebox left a comment

youjiali1995 left a comment

Yisaer left a comment

Yisaer Jun 3, 2021

JmPotato Jun 3, 2021 •

edited

Loading

JmPotato commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

Yisaer commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

store: refine the error handling and retry mechanism for stale read #24956

store: refine the error handling and retry mechanism for stale read #24956

Conversation

JmPotato commented May 28, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

JmPotato commented May 31, 2021

zhouqiang-cl commented May 31, 2021

JmPotato commented Jun 1, 2021

xhebox left a comment

Choose a reason for hiding this comment

Yisaer Jun 2, 2021

Choose a reason for hiding this comment

JmPotato Jun 2, 2021

Choose a reason for hiding this comment

Yisaer Jun 2, 2021

Choose a reason for hiding this comment

Yisaer left a comment

Choose a reason for hiding this comment

xhebox left a comment

Choose a reason for hiding this comment

JmPotato commented Jun 3, 2021

xhebox left a comment

Choose a reason for hiding this comment

youjiali1995 left a comment

Choose a reason for hiding this comment

Yisaer left a comment

Choose a reason for hiding this comment

Yisaer Jun 3, 2021

Choose a reason for hiding this comment

JmPotato Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

JmPotato commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

Yisaer commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

JmPotato commented May 28, 2021 •

edited

Loading

JmPotato Jun 3, 2021 •

edited

Loading