Introduce pre transfer leader #6539
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
BusyJay
added
sig/raft
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
hicqu
reviewed
Feb 7, 2020
src/raftstore/store/peer.rs
Outdated
| }; | ||
| if self.ready_to_transfer_leader(ctx, msg.get_index(), &from) { | ||
| self.transfer_leader(&from); | ||
| return; |
hicqu
reviewed
Feb 7, 2020
src/raftstore/store/peer.rs
Outdated
| fn pre_transfer_leader(&mut self, peer: &metapb::Peer) -> bool { | ||
| // Checks if safe to transfer leader. | ||
| if self.raft_group.raft.has_pending_conf() { | ||
| debug!( |
hicqu
reviewed
Feb 7, 2020
hicqu
reviewed
Feb 7, 2020
src/raftstore/store/peer.rs
Outdated
| if self.raft_group.raft.has_pending_conf() | ||
| || duration_to_sec(self.recent_conf_change_time.elapsed()) | ||
| < ctx.cfg.raft_reject_transfer_leader_duration.as_secs() as f64 | ||
| || self.raft_group.raft.pending_conf_index > index | ||
| { | ||
| debug!( |
There was a problem hiding this comment.
Is it better to change it to info?
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
hicqu
approved these changes
Feb 7, 2020
5kbpers
added
the
status/can-merge
|
/run-all-tests |
|
/bench +ycsb |
Benchmark Report
@@ Benchmark Diff @@
================================================================================
tidb: 17da140f150a2f800c03320e091d0ae2b9dba794
--- tikv: 1cbb91ecebce4c5f1b10c47323e4c806c38d5eb8
+++ tikv: 2968ca938b8f9cdaa6b6725833581a30ff7063e4
pd: bf4aba5f3f6e3433db2e319dd692e1215b6b88e7
================================================================================
read:
* OPS: 50882.53 ± 1.38% (std=488.00) delta: -1.54%
* Avg: 1.44 ± 1.94% delta: 2.01%
* p99: 8.00 ± 0.00% delta: 0.00%
update:
* OPS: 50883.75 ± 1.66% (std=573.79) delta: -1.59%
* Avg: 8.25 ± 2.48% delta: 1.00%
* p99: 16.00 ± 0.00% delta: 0.00%
|
|
cherry pick to release-3.0 failed |
|
cherry pick to release-3.1 failed |
BusyJay
added a commit
to BusyJay/tikv
that referenced
this pull request
Feb 8, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
This was referenced Feb 8, 2020
BusyJay
added a commit
to BusyJay/tikv
that referenced
this pull request
Feb 8, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
BusyJay
added a commit
to BusyJay/tikv
that referenced
this pull request
Feb 8, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
|
@BusyJay I believe this PR caused BR tests to almost always fail with "not leader" error (example: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/750/pipeline, search for "not leader" or "error 1"). Is there something we need to update in BR or TiDB or PD? cc @overvenus @3pointer (Edit: updating TiDB & PD deps to 4.0.0-beta doesn't help.) |
BusyJay
added a commit
that referenced
this pull request
Feb 10, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
AndreMouche
pushed a commit
that referenced
this pull request
Feb 10, 2020
youjiali1995
pushed a commit
that referenced
this pull request
Feb 14, 2020
solotzg
added a commit
to pingcap/tidb-engine-ext
that referenced
this pull request
Feb 15, 2020
* [3.1] external_storage: add S3 support (tikv#6209) (tikv#6536) * external_storage: add S3 support (tikv#6209) Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: kennytm <kennytm@gmail.com> * lock_manager: more metrics (tikv#6392) (tikv#6422) (tikv#6444) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * lock_manager: update default config (tikv#6426) (tikv#6429) (tikv#6446) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * Update rocksdb and titan. Update rust-rocksdb url (tikv#6554) Signed-off-by: Yi Wu <yiwu@pingcap.com> * raft_client: connect to other stores using peer_address (tikv#5569) (tikv#6491) * *: pick threads statistics for regions (tikv#6605) Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: disksing <i@disksing.com> * fix test_replica_read::test_read_after_cleanup_range_for_snap (tikv#6493) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * deadlock: more solid role change observer (tikv#6415) (tikv#6431) (tikv#6445) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * raftstore: read index message count metric (tikv#6579) (tikv#6610) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * raftstore: speed up conf change (tikv#6421) (tikv#6432) Signed-off-by: Jay Lee <busyjaylee@gmail.com> * raftstore: fix a panic in read index queue (tikv#6609) (tikv#6613) Signed-off-by: qupeng <qupeng@pingcap.com> * pick tikv#6414, tikv#6487, tikv#6477 and tikv#6539 to release-3.1 (tikv#6564) * raftstore: extract batch system (tikv#6414) * apply: support yield (tikv#6487) * batch-system: add benchmark (tikv#6477) * Introduce pre transfer leader (tikv#6539) Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set wait_merge_state to none after resuming pending state (tikv#6621) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * raftstore: learner load merge target & fix a merge network recovery bug (tikv#6623) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * change merge flow path (tikv#6617) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: Jay <BusyJay@users.noreply.github.com> Co-authored-by: gengliqi <gengliqiii@gmail.com>
solotzg
added a commit
to pingcap/tidb-engine-ext
that referenced
this pull request
Feb 21, 2020
* [3.1] external_storage: add S3 support (tikv#6209) (tikv#6536) * external_storage: add S3 support (tikv#6209) Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: kennytm <kennytm@gmail.com> * lock_manager: more metrics (tikv#6392) (tikv#6422) (tikv#6444) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * lock_manager: update default config (tikv#6426) (tikv#6429) (tikv#6446) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * Update rocksdb and titan. Update rust-rocksdb url (tikv#6554) Signed-off-by: Yi Wu <yiwu@pingcap.com> * raft_client: connect to other stores using peer_address (tikv#5569) (tikv#6491) * *: pick threads statistics for regions (tikv#6605) Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: disksing <i@disksing.com> * fix test_replica_read::test_read_after_cleanup_range_for_snap (tikv#6493) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * deadlock: more solid role change observer (tikv#6415) (tikv#6431) (tikv#6445) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * raftstore: read index message count metric (tikv#6579) (tikv#6610) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * raftstore: speed up conf change (tikv#6421) (tikv#6432) Signed-off-by: Jay Lee <busyjaylee@gmail.com> * raftstore: fix a panic in read index queue (tikv#6609) (tikv#6613) Signed-off-by: qupeng <qupeng@pingcap.com> * pick tikv#6414, tikv#6487, tikv#6477 and tikv#6539 to release-3.1 (tikv#6564) * raftstore: extract batch system (tikv#6414) * apply: support yield (tikv#6487) * batch-system: add benchmark (tikv#6477) * Introduce pre transfer leader (tikv#6539) Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set wait_merge_state to none after resuming pending state (tikv#6621) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * raftstore: learner load merge target & fix a merge network recovery bug (tikv#6623) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * change merge flow path (tikv#6617) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: Jay <BusyJay@users.noreply.github.com> Co-authored-by: gengliqi <gengliqiii@gmail.com>
c1ay
pushed a commit
to c1ay/tikv
that referenced
this pull request
May 9, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
ti-chi-bot
pushed a commit
that referenced
this pull request
Jan 29, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com> ### What problem does this PR solve? Problem Summary: The meaning of `leader-transfer-max-log-lag` has already been changed by #6539. For now, it means the difference value between the leader's last log index and the follower's last applied index. On the basis of implementation, we get the follower's last applied index through an RPC then we get the leader's last log index. There are many gaps between this process. 1. the applied index in peer fsm may be smaller than the real one in its corresponding apply fsm 2. the network latency 3. some logs may be proposed after sending the query RPC to this follower So the follower's last applied index may be much larger when we calculate the difference value. It thus appears that the default value of `leader-transfer-max-log-lag`(10) is too small which leads to many failures of leader transfer. I test the value of 128 and the failure is much less than before. * I test 256 and there is almost no failure. But maybe it's too large. Maybe we should find a more scientific approach to solve this problem later. ### What is changed and how it works? What's Changed: Change the default value of `leader-transfer-max-log-lag` to 128. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch Tests <!-- At least one of them must be included. --> - No code Side effects - No ### Release note <!-- bugfixes or new feature need a release note --> * Change the default `leader-transfer-max-log-lag` to 128 to increase the success rate of leader transfer
Little-Wallace
added a commit
to Little-Wallace/tikv
that referenced
this pull request
Feb 4, 2021
commit 74809fa Author: Drumil Patel <42701709+weastel@users.noreply.github.com> Date: Tue Feb 2 11:57:45 2021 +0530 copr: Vectorize AddDatetimeAndString (tikv#9610) Signed-off-by: Drumil Patel <drumilpatel720@gmail.com> <!-- Thank you for contributing to TiKV! If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document. If you're unsure about anything, just ask; somebody should be along to answer within a day or two. PR Title Format: 1. module [, module2, module3]: what's changed 2. *: what's changed If you want to open the **Challenge Program** pull request, please use the following template: https://raw.githubusercontent.com/tikv/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md You can use it with query parameters: https://github.com/tikv/tikv/compare/master...${you branch}?template=challenge-program.md --> ### What problem does this PR solve? Issue Number: Partially fixes tikv#9016 <!-- REMOVE this line if no issue to close --> Problem Summary: Added Vectorize version for AddDatetimeAndString ### What is changed and how it works? What's Changed: Implmented Vectorize vesion for AddDatetimeAndString ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test ### Release note <!-- bugfixes or new feature need a release note --> - Partially fixes tikv#9016 commit 103f5ff Author: Xintao <hunterlxt@live.com> Date: Tue Feb 2 13:13:45 2021 +0800 Make TiKV not to create dict files when master_key cannot work (tikv#9540) Signed-off-by: Xintao <hunterlxt@live.com> <!-- Thank you for contributing to TiKV! If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document. If you're unsure about anything, just ask; somebody should be along to answer within a day or two. PR Title Format: 1. module [, module2, module3]: what's changed 2. *: what's changed If you want to open the **Challenge Program** pull request, please use the following template: https://raw.githubusercontent.com/tikv/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md You can use it with query parameters: https://github.com/tikv/tikv/compare/master...${you branch}?template=challenge-program.md --> ### What problem does this PR solve? Issue Number: close tikv#9488 Problem Summary: At the first time TiKV access AWS KMS failed, it leaves corrupted dict file which causes TiKV to crash continuously. ### What is changed and how it works? check the `master_key` before creating the dicts. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test - Integration test - Manual test (add detailed scripts or steps below) - No code Side effects - Performance regression - Consumes more CPU - Consumes more MEM - Breaking backward compatibility ### Release note <!-- bugfixes or new feature need a release note --> - Fix the bug that TiKV could not start normally after failing to access AWS KMS. commit af4432f Author: Wallace <bupt2013211450@gmail.com> Date: Tue Feb 2 11:49:45 2021 +0800 importer: fix test_ingest_sst (tikv#9617) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> <!-- Thank you for contributing to TiKV! If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document. If you're unsure about anything, just ask; somebody should be along to answer within a day or two. PR Title Format: 1. module [, module2, module3]: what's changed 2. *: what's changed If you want to open the **Challenge Program** pull request, please use the following template: https://raw.githubusercontent.com/tikv/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md You can use it with query parameters: https://github.com/tikv/tikv/compare/master...${you branch}?template=challenge-program.md --> ### What problem does this PR solve? Issue Number: close tikv#9472 Problem Summary: TiKV may cleanup sst which has been upload just, so the ingest-job will success because it can not found file in path before it starts proposing, and the error is checked only when applied this entries. ### What is changed and how it works? We do not need to check and cleanup sst very quickly in this tests, so I remove the configuration for `cleanup_import_sst_interval`. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test - Integration test - Manual test (add detailed scripts or steps below) - No code Side effects - Performance regression - Consumes more CPU - Consumes more MEM - Breaking backward compatibility ### Release note <!-- bugfixes or new feature need a release note --> - No release note. commit 80582cb Author: wangggong <793160615@qq.com> Date: Mon Feb 1 13:15:45 2021 +0800 copr: Implement regexp & regexp_utf8 (tikv#9238) Signed-off-by: wangggong <793160615@qq.com> <!-- Thank you for contributing to TiKV! If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document. If you're unsure about anything, just ask; somebody should be along to answer within a day or two. PR Title Format: 1. module [, module2, module3]: what's changed 2. *: what's changed If you want to open the **Challenge Program** pull request, please use the following template: https://raw.githubusercontent.com/tikv/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md You can use it with query parameters: https://github.com/tikv/tikv/compare/master...${you branch}?template=challenge-program.md --> ### What problem does this PR solve? Issue Number: tikv#9016 Problem Summary: ### What is changed and how it works? What's Changed: Add regexp & regexp_utf8 ### Related changes ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test Side effects ### Release note <!-- bugfixes or new feature need a release note --> No release note. commit 7b260d1 Author: Jay <BusyJay@users.noreply.github.com> Date: Fri Jan 29 21:13:44 2021 +0800 resolve: detect tombstone correctly (tikv#9593) <!-- Thank you for contributing to TiKV! If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document. If you're unsure about anything, just ask; somebody should be along to answer within a day or two. PR Title Format: 1. module [, module2, module3]: what's changed 2. *: what's changed If you want to open the **Challenge Program** pull request, please use the following template: https://raw.githubusercontent.com/tikv/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md You can use it with query parameters: https://github.com/tikv/tikv/compare/master...${you branch}?template=challenge-program.md --> ### What problem does this PR solve? Issue Number: close tikv#9590 Problem Summary: PD client filter tombstone store and return error instead. Resolver should recognize the error and handle it correctly. ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test - Integration test ### Release note <!-- bugfixes or new feature need a release note --> - Fix repeated tombstone logs when sunset nodes commit a2c7aac Author: Liqi Geng <gengliqiii@gmail.com> Date: Fri Jan 29 04:03:44 2021 -0600 raftstore: enlarge leader-transfer-max-log-lag (tikv#9592) Signed-off-by: gengliqi <gengliqiii@gmail.com> ### What problem does this PR solve? Problem Summary: The meaning of `leader-transfer-max-log-lag` has already been changed by tikv#6539. For now, it means the difference value between the leader's last log index and the follower's last applied index. On the basis of implementation, we get the follower's last applied index through an RPC then we get the leader's last log index. There are many gaps between this process. 1. the applied index in peer fsm may be smaller than the real one in its corresponding apply fsm 2. the network latency 3. some logs may be proposed after sending the query RPC to this follower So the follower's last applied index may be much larger when we calculate the difference value. It thus appears that the default value of `leader-transfer-max-log-lag`(10) is too small which leads to many failures of leader transfer. I test the value of 128 and the failure is much less than before. * I test 256 and there is almost no failure. But maybe it's too large. Maybe we should find a more scientific approach to solve this problem later. ### What is changed and how it works? What's Changed: Change the default value of `leader-transfer-max-log-lag` to 128. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch Tests <!-- At least one of them must be included. --> - No code Side effects - No ### Release note <!-- bugfixes or new feature need a release note --> * Change the default `leader-transfer-max-log-lag` to 128 to increase the success rate of leader transfer commit 3a4b8bb Author: Greg Weber <greg@pingcap.com> Date: Thu Jan 28 20:41:44 2021 -0600 fix: broken build on Mac (tikv#9598) On Mac: ``` --> components/file_system/src/rate_limiter.rs:5:5 5 | use crossbeam_utils::CachePadded; | ^^^^^^^^^^^^^^^ use of undeclared crate or module `crossbeam_utils` ``` ### Release note * None commit 20b0e95 Author: Xinye Tao <xy.tao@outlook.com> Date: Thu Jan 28 18:05:43 2021 +0800 engine: refactor `MetricsFlusher` and add fallback storage IO measuring (tikv#9458) Signed-off-by: tabokie <xy.tao@outlook.com> ### What is changed and how it works? What's Changed: Remove metrics flusher in engine_traits and use background worker instead, add IO metrics task which reports statistics from iosnooper, or IO rate limiter if BCC is unavailable. ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test ### Release note <!-- bugfixes or new feature need a release note --> No release note Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What have you changed?
This is an implement of tikv/rfcs#37.
The implement reuses TransferLeader message of raft to avoid introduce extra
proto message. Note that we don't send TransferLeader message to the network
before, hence the message can only be sent out from a new leader/follower.
If an old follower receives the message, it will just redirect back to the leader,
which will ignore the message as it has an invalid index.
What is the type of the changes?
How is the PR tested?
Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No.
Does this PR affect
tidb-ansible?No.
Refer to a related PR or issue link
tikv/rfcs#37