codec: add compile time assert to guarantee safety #11357

BusyJay · 2021-11-12T08:57:03Z

What problem does this PR solve?

Issue Number: close #11356.

Check List

Tests

Unit test
Integration test

Release note

None

close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

ti-chi-bot · 2021-11-12T08:57:04Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

breeswish
skyzh

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

skyzh

Why is this a compile-time check? It seems that padding_size is calculated by let padding_size = T::parse_padding_size(*src_ptr.add(MEMCMP_GROUP_SIZE));, which can only be known at runtime.

breezewish · 2021-11-12T10:34:00Z

components/codec/src/byte.rs

@@ -372,7 +367,8 @@ impl MemComparableByteCodec {
                    // is faster than checking pad bytes one by one, since it will compare multiple
                    // bytes at once.
                    let base_padding_ptr = dest_ptr.sub(padding_size);
-                    let expected_padding_ptr = T::get_raw_padding_ptr();
+                    // Force a compile time check to ensure safety. The check will be optimized away.
+                    let expected_padding_ptr = T::PADDING[..padding_size].as_ptr();


padding_size is not a constant but a variable, how would compiler discover that there is out of bound possibility?

Because L362 has checked that padding_size <= MEMCMP_GROUP_SIZE, and the len of T::PADDING is exact MEMCMP_GROUP_SIZE. /cc @skyzh

This sounds reasonable to me.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

skyzh · 2021-11-16T04:10:08Z

/merge

ti-chi-bot · 2021-11-16T04:10:09Z

@skyzh: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-11-16T04:10:11Z

This pull request has been accepted and is ready to merge.

Commit hash: 2c3eeb7

BusyJay · 2021-11-17T08:56:14Z

/test

The failed case is traced at #11365.

ti-chi-bot · 2021-11-17T09:21:33Z

@BusyJay: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

# This is the 1st commit message: Add latency stats for kv get and kv batch get Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message #2: tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <kennytm@gmail.com> # This is the commit message tikv#3: codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add more explanation. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#4: rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#5: backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#6: Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix test error Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#7: ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#8: Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#9: gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * avoid using prefix seek Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename test Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix build Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#10: fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <qupeng@pingcap.com> * close tikv#11353 Signed-off-by: qupeng <qupeng@pingcap.com> * address comments Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#11: build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <xy.tao@outlook.com> * fix cargo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#12: engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <xy.tao@outlook.com> * use constant Signed-off-by: tabokie <xy.tao@outlook.com> * fix typo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#13: gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <zbk602423539@gmail.com> # This is the commit message tikv#14: raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <gengliqiii@gmail.com> * add tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove offset in NormalFsm Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#15: metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#16: resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * more log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#17: metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#18: storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#19: test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <1045931706@qq.com> * make test statable Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * close tikv#11365 Signed-off-by: bufferflies <1045931706@qq.com> * reformat annotation Signed-off-by: bufferflies <1045931706@qq.com> * add send once filter Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * replace sendOnceMsgFilter Signed-off-by: bufferflies <1045931706@qq.com> * remove unnecessary change Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#20: raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * do not change seek Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix test Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix number overflow Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * revert time Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * log large range gc task Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * address comment Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * close tikv#11404 Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#21: Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#22: Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> # This is the commit message tikv#23: raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#24: fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <zhengxiaojin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#25: resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close tikv#11400 Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#26: support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#27: copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <you1474600@gmail.com> * switch to new kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bugs... Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> * remove unsed field Signed-off-by: you06 <you1474600@gmail.com> * add comments Signed-off-by: you06 <you1474600@gmail.com> * update kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bench Signed-off-by: you06 <you1474600@gmail.com> * update cargo.lock Signed-off-by: you06 <you1474600@gmail.com> * update comment Signed-off-by: you06 <you1474600@gmail.com> * add test for copr paging Signed-off-by: you06 <you1474600@gmail.com> * close tikv#11448 Signed-off-by: you06 <you1474600@gmail.com> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Signed-off-by: you06 <you1474600@gmail.com> * check range in response Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test for reverse scan Signed-off-by: you06 <you1474600@gmail.com> * add test for multi ranges Signed-off-by: you06 <you1474600@gmail.com> * test drained with multi ranges Signed-off-by: you06 <you1474600@gmail.com> * fix clippy Signed-off-by: you06 <you1474600@gmail.com> * fix lint Signed-off-by: you06 <you1474600@gmail.com> * paging by all record count Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> # This is the commit message tikv#28: encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <xy.tao@outlook.com> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix error from merge Signed-off-by: tabokie <xy.tao@outlook.com> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * add wrappers Signed-off-by: tabokie <xy.tao@outlook.com> * remove pub quantifiers and fix lint Signed-off-by: tabokie <xy.tao@outlook.com> * fix name export Signed-off-by: tabokie <xy.tao@outlook.com> * revert naming changes Signed-off-by: tabokie <xy.tao@outlook.com> * address comment on unit test Signed-off-by: tabokie <xy.tao@outlook.com> * fix test and inline wrapper calls Signed-off-by: tabokie <xy.tao@outlook.com> * fix comments Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#29: server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <rleungx@gmail.com> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <rleungx@gmail.com> # This is the commit message tikv#30: engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * overhaul configurations Signed-off-by: tabokie <xy.tao@outlook.com> * update default config and address comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#31: *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#32: coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#33: server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#34: raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#35: engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <xy.tao@outlook.com> * handle timeout error Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#36: metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#37: *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * storage: scan support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * update kvproto and add some comments Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * test desc scan Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#38: storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <andylokandy@hotmail.com> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * remove println Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * disable test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add case Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#39: *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> # This is the commit message tikv#40: Add comments Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#41: Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#42: raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable transfer leader case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> # This is the commit message tikv#43: engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * introduce purge-worker Signed-off-by: tabokie <xy.tao@outlook.com> * clean up old worker Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#44: Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * fix unit test Signed-off-by: dwangxxx <dwang_slam@163.com> * fix format Signed-off-by: dwangxxx <dwang_slam@163.com> * fix test Signed-off-by: dwangxxx <dwang_slam@163.com> * add test. Signed-off-by: dwangxxx <dwang_slam@163.com> * fix clippy error Signed-off-by: dwangxxx <dwang_slam@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#45: Rename KvGetLatencyStats Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#46: Rename Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#47: *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#48: cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#49: metric: add stale read count metric (tikv#11051) Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#50: storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <lpbgytong@163.com> # This is the commit message tikv#51: storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#52: raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#53: scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#54: *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#55: backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <yujuncen@pingcap.com> # This is the commit message tikv#56: tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#57: fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <huangzhe8263@gmail.com> * reformat by lints Signed-off-by: onlyacat <huangzhe8263@gmail.com> * close tikv#11552 Signed-off-by: onlyacat <huangzhe8263@gmail.com> # This is the commit message tikv#58: raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add log for becoming follower Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * correct commit log check Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Signed-off-by: mornyx <mornyx.z@gmail.com> close tikv#8942 tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <kennytm@gmail.com> codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add more explanation. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * format Signed-off-by: gengliqi <gengliqiii@gmail.com> backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix test error Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <qupeng@pingcap.com> gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * avoid using prefix seek Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename test Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix build Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <qupeng@pingcap.com> * close tikv#11353 Signed-off-by: qupeng <qupeng@pingcap.com> * address comments Signed-off-by: qupeng <qupeng@pingcap.com> build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <xy.tao@outlook.com> * fix cargo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <xy.tao@outlook.com> * use constant Signed-off-by: tabokie <xy.tao@outlook.com> * fix typo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <zbk602423539@gmail.com> raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <gengliqiii@gmail.com> * add tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove offset in NormalFsm Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <xy.tao@outlook.com> resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * more log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <1045931706@qq.com> * make test statable Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * close tikv#11365 Signed-off-by: bufferflies <1045931706@qq.com> * reformat annotation Signed-off-by: bufferflies <1045931706@qq.com> * add send once filter Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * replace sendOnceMsgFilter Signed-off-by: bufferflies <1045931706@qq.com> * remove unnecessary change Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * do not change seek Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix test Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix number overflow Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * revert time Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * log large range gc task Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * address comment Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * close tikv#11404 Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <zhengxiaojin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close tikv#11400 Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <you1474600@gmail.com> * switch to new kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bugs... Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> * remove unsed field Signed-off-by: you06 <you1474600@gmail.com> * add comments Signed-off-by: you06 <you1474600@gmail.com> * update kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bench Signed-off-by: you06 <you1474600@gmail.com> * update cargo.lock Signed-off-by: you06 <you1474600@gmail.com> * update comment Signed-off-by: you06 <you1474600@gmail.com> * add test for copr paging Signed-off-by: you06 <you1474600@gmail.com> * close tikv#11448 Signed-off-by: you06 <you1474600@gmail.com> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Signed-off-by: you06 <you1474600@gmail.com> * check range in response Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test for reverse scan Signed-off-by: you06 <you1474600@gmail.com> * add test for multi ranges Signed-off-by: you06 <you1474600@gmail.com> * test drained with multi ranges Signed-off-by: you06 <you1474600@gmail.com> * fix clippy Signed-off-by: you06 <you1474600@gmail.com> * fix lint Signed-off-by: you06 <you1474600@gmail.com> * paging by all record count Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <xy.tao@outlook.com> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix error from merge Signed-off-by: tabokie <xy.tao@outlook.com> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * add wrappers Signed-off-by: tabokie <xy.tao@outlook.com> * remove pub quantifiers and fix lint Signed-off-by: tabokie <xy.tao@outlook.com> * fix name export Signed-off-by: tabokie <xy.tao@outlook.com> * revert naming changes Signed-off-by: tabokie <xy.tao@outlook.com> * address comment on unit test Signed-off-by: tabokie <xy.tao@outlook.com> * fix test and inline wrapper calls Signed-off-by: tabokie <xy.tao@outlook.com> * fix comments Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <rleungx@gmail.com> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <rleungx@gmail.com> engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * overhaul configurations Signed-off-by: tabokie <xy.tao@outlook.com> * update default config and address comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <xy.tao@outlook.com> coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <xy.tao@outlook.com> * handle timeout error Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * storage: scan support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * update kvproto and add some comments Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * test desc scan Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <andylokandy@hotmail.com> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * remove println Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * disable test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add case Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Add comments Signed-off-by: mornyx <mornyx.z@gmail.com> Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <mornyx.z@gmail.com> raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable transfer leader case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * introduce purge-worker Signed-off-by: tabokie <xy.tao@outlook.com> * clean up old worker Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * fix unit test Signed-off-by: dwangxxx <dwang_slam@163.com> * fix format Signed-off-by: dwangxxx <dwang_slam@163.com> * fix test Signed-off-by: dwangxxx <dwang_slam@163.com> * add test. Signed-off-by: dwangxxx <dwang_slam@163.com> * fix clippy error Signed-off-by: dwangxxx <dwang_slam@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Rename KvGetLatencyStats Signed-off-by: mornyx <mornyx.z@gmail.com> Rename Signed-off-by: mornyx <mornyx.z@gmail.com> *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com> cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <qupeng@pingcap.com> metric: add stale read count metric (tikv#11051) Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <lpbgytong@163.com> storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <yujuncen@pingcap.com> tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <qupeng@pingcap.com> fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <huangzhe8263@gmail.com> * reformat by lints Signed-off-by: onlyacat <huangzhe8263@gmail.com> * close tikv#11552 Signed-off-by: onlyacat <huangzhe8263@gmail.com> raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add log for becoming follower Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * correct commit log check Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage: Add API V2 check for RawKV and TxnKV requests (part 2) (tikv#11228) * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * resolve conflict Signed-off-by: pingyu <yuping@pingcap.com> * add raw_key_maybe_unbounded_into_option Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: address comments Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * remove make_invalid_key_prefix_err Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: andylokandy <andylokandy@hotmail.com> storage: read and write in-memory pessimistic locks in the scheduler (tikv#11529) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> tolerate unmounted cgroup controller (tikv#11570) * tolerate unmounted cgroup controller, fix tikv#11569 Signed-off-by: tabokie <xy.tao@outlook.com> * add test Signed-off-by: tabokie <xy.tao@outlook.com> * address comment Signed-off-by: tabokie <xy.tao@outlook.com> cdc: stabilize some cdc cases (tikv#11586) * close tikv#11560 Signed-off-by: qupeng <qupeng@pingcap.com>

codec: add compile time assert to guarantee safety

00d6834

close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

BusyJay added the type/enhancement The issue or PR belongs to an enhancement. label Nov 12, 2021

BusyJay requested review from breezewish and skyzh November 12, 2021 08:57

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 12, 2021

skyzh reviewed Nov 12, 2021

View reviewed changes

breezewish reviewed Nov 12, 2021

View reviewed changes

add more explanation.

2c3eeb7

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

skyzh approved these changes Nov 13, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 13, 2021

breezewish approved these changes Nov 16, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 16, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 16, 2021

Merge branch 'master' into correctly-declare-safety

4928b58

Merge branch 'master' into correctly-declare-safety

57f52dd

ti-chi-bot merged commit 1ea4920 into tikv:master Nov 17, 2021

BusyJay deleted the correctly-declare-safety branch November 17, 2021 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

codec: add compile time assert to guarantee safety #11357

codec: add compile time assert to guarantee safety #11357

BusyJay commented Nov 12, 2021

ti-chi-bot commented Nov 12, 2021 •

edited

Loading

skyzh left a comment

breezewish Nov 12, 2021

BusyJay Nov 12, 2021

skyzh Nov 13, 2021

skyzh commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

BusyJay commented Nov 17, 2021

ti-chi-bot commented Nov 17, 2021

codec: add compile time assert to guarantee safety #11357

codec: add compile time assert to guarantee safety #11357

Conversation

BusyJay commented Nov 12, 2021

What problem does this PR solve?

Check List

Release note

ti-chi-bot commented Nov 12, 2021 • edited Loading

skyzh left a comment

Choose a reason for hiding this comment

breezewish Nov 12, 2021

Choose a reason for hiding this comment

BusyJay Nov 12, 2021

Choose a reason for hiding this comment

skyzh Nov 13, 2021

Choose a reason for hiding this comment

skyzh commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

BusyJay commented Nov 17, 2021

ti-chi-bot commented Nov 17, 2021

ti-chi-bot commented Nov 12, 2021 •

edited

Loading