Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge master about 6.6(7ec73fd) #262

Merged
merged 41 commits into from
Jan 31, 2023

Conversation

CalvinNeo
Copy link
Member

@CalvinNeo CalvinNeo commented Jan 29, 2023

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

This PR is to adopt Flashback fix:

  1. First merge 9726e5. No flashback fix included, do this because we want less work when we actually merge flashback fix.
  2. Then merge 7ec73fd. This is flashback fix.
    Origin 453b9df.

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

ethercflow and others added 30 commits December 28, 2022 12:00
ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
Signed-off-by: Zwb <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

a few panic fix
1) update_approximate_raft_log_size may run into divid by zero error 
2) appy_delete may have None write_batch
3) StoreMeta::set_region may run into region corruption error if it's destroyed and re-created. 
4) TabletSnapManager's snapshot size calculation may throw Other error.

Signed-off-by: qi.xu <tonxuqi@outlook.com>
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
Co-authored-by: Jay Lee <BusyJayLee@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…13995)

ref tikv#12842

Whenever timeout, the peer will check for unapplied logs whether
there are pending conf change and trigger heavy reads. So we
wait till most logs are applied before ticking.

It also fix following issues:
- PersistenceListener is not installed
- implementation of persisted_apply_index is wrong
- parse tablet name is wrong

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
ref tikv#12842

Signed-off-by: bufferflies <1045931706@qq.com>
close tikv#13997

Support to use evict_entry_cache when restart node.

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: hongyunyan <649330952@qq.com>
Signed-off-by: Xinye Tao <xy.tao@outlook.com>
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
Signed-off-by: Zwb <ethercflow@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Jay <BusyJay@users.noreply.github.com>
Co-authored-by: Zwb <ethercflow@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

The API is supposed to be used with `append` but nowhere can we find
the clue. This PR merges `cut_logs` and `append` to reduce confusion
and mistakes.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

Publish tablet in apply thread is unsafe. This PR moves the operation to
raftstore. It also fixes the issues that applying two splits at a time can
cause panic. It also makes sure cache will be cleared after tablet is published.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
ref tikv#12842

These two are helpers to utilize the static KV pairs in logger. In the
past, we use `logger.list()` to try to format the configured KV pairs,
but it will not work as values are omitted.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#13730

Introduce priority-based channel

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

When the tablet contains dirty data right after split, generating snapshot may
just a waste. On the other hand, split usually happens on all peers, so delay
it a bit actually makes all peers more likely to be initialized by split. So
this PR rejects generating snapshot when it detects it still has dirty data.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
ref tikv#12842

1. store heartbeat should add snapshot and kv engine used size

Signed-off-by: bufferflies <1045931706@qq.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
close tikv#13977

Signed-off-by: tabokie <xy.tao@outlook.com>
ref tikv#12842

Make apply adaptive to reduce high tail latency.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

- add water metrics
- fix potential panic when destroying a peer
- fix incorrect store size

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
* util: Fix incorrect memory capacity

Signed-off-by: Wish <breezewish@outlook.com>

* Fix lints

Signed-off-by: Wish <breezewish@outlook.com>

* Check capacity with /proc/meminfo

Signed-off-by: Wish <breezewish@outlook.com>

Signed-off-by: Wish <breezewish@outlook.com>
* hotfix kvproto for global config

Signed-off-by: husharp <jinhao.hu@pingcap.com>

* make format happy

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Signed-off-by: husharp <jinhao.hu@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#13730

Signed-off-by: glorv <glorvs@163.com>
…ock request (tikv#14037)

close tikv#14038, close pingcap/tidb#40114

Fixes the problem that when handling repeated acquire_pessimistic_lock requests is recevied, should_not_exist is ignored. 

TiKV provides idempotency for these RPC requests, but for acquire_pessimistic_lock, it ignored the possibility that the client may expect a pessimistic_rollback between two acquire_pessimistic_lock request on the same key. In this case the second request may come from another statement and carries `should_not_exist` that wasn't set in the previously finished pessimistic lock request. If the first request successfully acquired the lock and the pessimistic_rollback failed, TiKV may return a sucessful response, making the client believe that the key doesn't exist before. In some rare cases, this has risk to cause data inconsistency.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

This PR fixes several bugs and metrics:
- Now waterfall timer will be reset in before_write, the goal is to solve
    the confusion that stall writes can pollute the whole waterfall metrics.
- Perf context is changed not to be associated with engine instance. Perf
      context is thread local and instance independent under the hook.
- Fix flushed index advance failure due to suspicious flush.
- Support print long uncommitted logs and fix incorrect commit time

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#12842

Move transaction related code to txn_ext.rs.

Fix the bug that snapshot doesn't set term and extra_op.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
ref tikv#12842

1) add snapshot apply metrics
2) disable bloomfilter for raftkv-v2 for now until a proper ratio is found 
3) disable rocksdb write stall for raftkv-v2 until the tablet flow control is fully verified.

Signed-off-by: Qi Xu <tonyxuqqi@outlook.com>

Co-authored-by: Qi Xu <tonyxuqqi@outlook.com>
ref tikv#12876

fix witness raft log gc panic and refactor

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
ref tikv#12999

copr: support handling keyspace request

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>
ref tikv#13730

Support priority-based scheduling for the scheduler worker pool.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
ref tikv#11312

Fix `make docker`.

Signed-off-by: tabokie <xy.tao@outlook.com>
close tikv#14063

make sure kvproto compatibility

Signed-off-by: husharp <jinhao.hu@pingcap.com>
ref tikv#12842

Signed-off-by: tabokie <xy.tao@outlook.com>
close tikv#14071

Log backup would aware TLS certifications changing.

Signed-off-by: hillium <yujuncen@pingcap.com>
close tikv#14069

Signed-off-by: YangKeao <yangkeao@chunibyo.icu>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ethercflow and others added 4 commits January 19, 2023 22:15
* support switch witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* add switch witness api for test_pd_client

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* pd heartbeat resp support switch witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update region epoch

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* fix write apply state race

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* remove unnecessary code

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* add back test_witness_conf_change

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* add some tests

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* avoid test failures

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* a few refactor

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* add witness election priority and address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* clean code

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* fix tests failed caused by cfg

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* fix test failed caused by mistake modify

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* adjust priority after snapshot persisted

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* notify pd after switch witness as region changed

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* define a new backoff error for witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* fix panic caused by applygap

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* forbid transfer leader to non-witness waiting data

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update kvproto

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* fix two panics

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* retry request snapshot

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* retry to request snaphost after term change

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update kvproto comment

ref tikv#12876

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
Signed-off-by: Zwb <ethercflow@gmail.com>
Co-authored-by: Xinye Tao <xy.tao@outlook.com>
close tikv#13983

- add etcd mock for pd
- add service for resource group

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ref tikv#13730

Support priority-based scheduling for the apply batch system.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@CLAassistant
Copy link

CLAassistant commented Jan 29, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
7 out of 18 committers have signed the CLA.

✅ hongyunyan
✅ HuSharp
✅ glorv
✅ CalvinNeo
✅ YangKeao
✅ breezewish
✅ LittleFall
❌ ethercflow
❌ tonyxuqqi
❌ BusyJay
❌ Connor1996
❌ tabokie
❌ bufferflies
❌ MyonKeminta
❌ iosmanthus
❌ YuJuncen
❌ andreid-db
❌ JmPotato
You have signed the CLA already but the status is still pending? Let us recheck it.

andreid-db and others added 3 commits January 29, 2023 12:15
close tikv#13586, close tikv#13752, ref tikv#14017

Signed-off-by: Andrei Dragus <andrei.dragus@databricks.com>
…#14087)

close tikv#14086

PD_REQUEST_HISTOGRAM_VEC can be changed to use static metrics to improve performance.

Signed-off-by: husharp <jinhao.hu@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
@CalvinNeo CalvinNeo changed the title Merge tikv release 6.6(9726e5) Merge master about 6.6(9726e5) Jan 29, 2023
LittleFall and others added 4 commits January 30, 2023 17:25
…ikv#14088)

close tikv#14045

- Store the flashback `start_ts` in region meta.
- Allow the read request with a smaller ts during flashback.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…ata. (tikv#14046)

ref tikv#12999

import: sst_importer support download SST and rewrite into keyspace data.

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>
@CalvinNeo CalvinNeo changed the title Merge master about 6.6(9726e5) Merge master about 6.6(7ec73fd) Jan 31, 2023
@CalvinNeo CalvinNeo merged commit 6444536 into pingcap:raftstore-proxy Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.