Depend TiKV as a component #177

CalvinNeo · 2022-09-22T12:35:59Z

What problem does this PR solve?

Issue Number: close pingcap/tiflash#5170

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ref tikv#12602, close tikv#12670 Signed-off-by: glorv <glorvs@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close tikv#12680, ref tikv#12680 Fix issue tikv#12680. Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: iosmanthus <dengliming@pingcap.com>

close tikv#12698 Close engine before cleaning up its data during raft engine migration. Signed-off-by: tabokie <xy.tao@outlook.com>

close tikv#12709 Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

close tikv#12702 Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>

…n check is needed and scan is used for it. (tikv#12678) ref tikv#12597 When bucket is enabled, using CheckPolicy::Scan would lead to large amount of read IO after tikv restart. Before this PR, the Scan is used unless the region size reaches 1.5G, which is very rare for a 96 ~ 256MB's region-split-size. After this change, generating bucket won't introduce new scan unless the scan is necessary for splitting region. This can significantly reduce the read IO. Also refine the logic for the fix of 12597. Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref tikv#12717 cdc: skip prewrite without value Signed-off-by: qupeng <qupeng@pingcap.com>

close tikv#12707 This PR optimize debug build by disabling all debuginfo excepts tests itself. So that the generated artifacts will be smaller and also speed up compile time a little. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref tikv#12592 cdc: make tso worker threads configuable Signed-off-by: qupeng <qupeng@pingcap.com>

close tikv#12711 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close tikv#12359 This commit makes use of the `schedule_time` in yatp to calculate the wait duration of each wake. The wait duration panel is added to the grafana for the unified read pool and the txn scheduler pool. Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

…#12127) ref tikv#11424, ref tikv#11840 Signed-off-by: tabokie <xy.tao@outlook.com>

ref tikv#12739, close tikv#12739 See tikv#12739 Signed-off-by: gengliqi <gengliqiii@gmail.com>

…ikv#12713) ref tikv#11612 Signed-off-by: ekexium <ekexium@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref tikv#12362 This commit measures the read duration spent on reading (e.g. write conflict checks). This fixes the missing part of scheduler commands and may help diagnosis when scheduler reading takes a long time. This commit also changes some now_coarse to now. now_coarse has a precision of 10ms on many systems, so it may not meet our precision requirement. Instant::now is fast enough to be called in these cases. Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close tikv#12758 Signed-off-by: haojinming <jinming.hao@pingcap.com>

…2757) close tikv#12750, close tikv#12751 Signed-off-by: joccau <zak.zhao@pingcap.com>

close tikv#12660 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

…v#12777) close tikv#12578 As the issue shows, try_batch may use out of date bucket meta for further operation which has not been detected by any test. This PR adds a test for it. Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close tikv#12771 Signed-off-by: 3pointer <luancheng@pingcap.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref tikv#12362 This commit is a large refactoring that moves collecting engine PerfContext from storage and coprocessor to engine_rocks and the tracker. Now, the storage and coprocessor are mostly decoupled with a specific engine (engine_rocks). And it introduces a general trakcer mechanism to collect the metrics of a request during its whole lifetime. It will help us collect more performance critical data of a single request more easily. Signed-off-by: Yilin Chen <sticnarf@gmail.com>

close tikv#12797 Deprecate gc-merge-rewrite option. Signed-off-by: tabokie <xy.tao@outlook.com>

close tikv#12773 replace signal with signal-hook and update other deps by depbot. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

close tikv#12780 Signed-off-by: glorv <glorvs@163.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

tikv#12756) ref tikv#12772 add tablet related function into Engines API. The idea is that for region's KV data read/write, using tablet() instead of engines.kv regardless multirocks is enabled or not. Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com> Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com>

close tikv#12803 In the investigation process, the debug log information helps a lot, but there's still some important information missing such as retrying or 1pc flags. Changed: 1. Add more useful information displaying commands. 2. Redact necessary parts. Signed-off-by: cfzjywxk <lsswxrxr@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

close tikv#12787 Add missing metrics for async io code path Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref tikv#11044, close tikv#11044 Signed-off-by: Lucasliang <nkcs_lykx@hotmail.com>

close tikv#12345 do not reconnect for pd unknown error Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ref rust-lang/cargo#10250, close tikv#12804, ref rust-lang/rust#68007, ref rust-lang/rust#96132 Signed-off-by: tabokie <xy.tao@outlook.com>

…store (tikv#13374) close tikv#13376 Signed-off-by: fengou1 <feng.ou@pingcap.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: fengou1 <85682690+fengou1@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com>

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

…ll-depend-tikv

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CLAassistant · 2022-09-22T12:36:18Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
9 out of 16 committers have signed the CLA.

✅ CalvinNeo
✅ Leavrth
✅ hehechen
✅ BornChanger
✅ mengxin9014
✅ YangKeao
✅ fengou1
✅ gengliqi
✅ SpadeA-Tang
❌ rleungx
❌ sticnarf
❌ tabokie
❌ 3pointer
❌ haojinming
❌ BusyJay
❌ 5kbpers
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

…nd-tikv

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CalvinNeo · 2022-10-31T09:43:00Z

In #210

glorv and others added 30 commits May 30, 2022 18:38

server: fix the bug that metrics data returns twice (tikv#12671)

d0bc9d1

ref tikv#12602, close tikv#12670 Signed-off-by: glorv <glorvs@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

causal_ts: fix issue tikv#12680 (tikv#12697)

8d1742c

close tikv#12680, ref tikv#12680 Fix issue tikv#12680. Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: iosmanthus <dengliming@pingcap.com>

server: fix race during raft data migration (tikv#12701)

cfd0b0e

close tikv#12698 Close engine before cleaning up its data during raft engine migration. Signed-off-by: tabokie <xy.tao@outlook.com>

log backup: disable test log by default (tikv#12710)

6a67b08

close tikv#12709 Signed-off-by: Yu Juncen <yujuncen@pingcap.com>

Add link to website of TiKV's creator (tikv#12703)

6d883b3

close tikv#12702 Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com>

cdc: skip prewrite without value (tikv#12612)

8a22454

ref tikv#12717 cdc: skip prewrite without value Signed-off-by: qupeng <qupeng@pingcap.com>

cdc: make tso worker threads configuable (tikv#12576)

4a8a3c5

ref tikv#12592 cdc: make tso worker threads configuable Signed-off-by: qupeng <qupeng@pingcap.com>

security, server: fix running local test (tikv#12712)

761d591

close tikv#12711 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

engine: only override write stall configurations if unspecified (tikv…

2a508a5

…#12127) ref tikv#11424, ref tikv#11840 Signed-off-by: tabokie <xy.tao@outlook.com>

copr: fix a wrong check in time parsing (tikv#12740)

da16e5e

ref tikv#12739, close tikv#12739 See tikv#12739 Signed-off-by: gengliqi <gengliqiii@gmail.com>

log details for PessimisitcLockNotFound in check_for_newer_version (t…

033d62d

…ikv#12713) ref tikv#11612 Signed-off-by: ekexium <ekexium@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Reserve key space id encoding in backup convert (tikv#12759)

62545b0

close tikv#12758 Signed-off-by: haojinming <jinming.hao@pingcap.com>

log-backup: fix bug about restoring point at TiCloud with KMS (tikv#1…

dee0e1e

…2757) close tikv#12750, close tikv#12751 Signed-off-by: joccau <zak.zhao@pingcap.com>

*: update jemalloc to 5.3.0 (tikv#12661)

ae46f9b

close tikv#12660 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

config: output warn log when some components config invalid (tikv#12767)

fd7b4ad

close tikv#12771 Signed-off-by: 3pointer <luancheng@pingcap.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

engine: deprecate gc-merge-rewrite option (tikv#12798)

c17e29b

close tikv#12797 Deprecate gc-merge-rewrite option. Signed-off-by: tabokie <xy.tao@outlook.com>

update several deps (tikv#12775)

265dbd2

close tikv#12773 replace signal with signal-hook and update other deps by depbot. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

server: check raft_client config change after flush (tikv#12781)

050b6c0

close tikv#12780 Signed-off-by: glorv <glorvs@163.com> Co-authored-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

metrics: Add missing metrics for async io code path (tikv#12788)

7bfcf60

close tikv#12787 Add missing metrics for async io code path Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

raftstore: reset perf context before each apply write (tikv#12808)

0d9b7b3

ref tikv#11044, close tikv#11044 Signed-off-by: Lucasliang <nkcs_lykx@hotmail.com>

pd_client: Do not reconnect for pd unknown error (tikv#12827)

11b5d4c

close tikv#12345 do not reconnect for pd unknown error Signed-off-by: Connor1996 <zbk602423539@gmail.com>

tests: ignore env var dependent tests at runtime (tikv#12805)

2fbf7ee

ref rust-lang/cargo#10250, close tikv#12804, ref rust-lang/rust#68007, ref rust-lang/rust#96132 Signed-off-by: tabokie <xy.tao@outlook.com>

fengou1 and others added 19 commits September 17, 2022 19:58

Merge remote-tracking branch 'tikv/master' into merge-to-tikv-6.3-master

251198c

fix encryption and pprof

399693f

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

fix fmt

67adb1d

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

part fix rocksdb

28e1db2

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

use most files from new engine_rocks

e96a81a

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

adapt engine_tiflash part 3

fc95ad0

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

recover tests

69f8d77

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

fix except from engine

2c4b059

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

try fix proxy_server

94c05cf

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

fix except rocksdb

4110892

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

fix lifetime problem for sst reader

ce173dd

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

clean

4deee6c

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

remove jemalloc

7871067

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

run tests

bfb33d5

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

remove compat_old_proxy

f7328fc

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

Merge remote-tracking branch 'origin/merge-to-tikv-6.3-master' into a…

67a406c

…ll-depend-tikv

try all change to git

94da964

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

fix crate tikv

1e20b59

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CalvinNeo mentioned this pull request Sep 22, 2022

Refactor TiFlash Proxy pingcap/tiflash#5170

Open

rev b5329ee

8295fba

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CalvinNeo requested a review from breezewish October 8, 2022 03:12

CalvinNeo added 4 commits October 8, 2022 11:27

Merge remote-tracking branch 'upstream/raftstore-proxy' into all-depe…

8833394

…nd-tikv

addr

f8c92cc

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

adapt 1

b9f45a0

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

adapt 2

1de47fe

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CalvinNeo mentioned this pull request Oct 31, 2022

Remove abundant tests and configs #208

Merged

12 tasks

CalvinNeo closed this Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Depend TiKV as a component #177

Depend TiKV as a component #177

CalvinNeo commented Sep 22, 2022

CLAassistant commented Sep 22, 2022 •

edited

Loading

CalvinNeo commented Oct 31, 2022

Depend TiKV as a component #177

Depend TiKV as a component #177

Conversation

CalvinNeo commented Sep 22, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented Sep 22, 2022 • edited Loading

CalvinNeo commented Oct 31, 2022

CLAassistant commented Sep 22, 2022 •

edited

Loading