Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Dynamic Region] Some large regions did not split as expected #12597

Closed
Lily2025 opened this issue May 20, 2022 · 3 comments · Fixed by #12620
Closed

[Dynamic Region] Some large regions did not split as expected #12597

Lily2025 opened this issue May 20, 2022 · 3 comments · Fixed by #12620
Assignees
@5kbpers
Labels
affects-6.1 severity/major type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented May 20, 2022 •
edited
Loading

Bug Report

What version of TiKV are you using?

./tikv-server -V
TiKV
Release Version: 6.1.0-alpha
Edition: Community
Git Commit Hash: a23fd68
Git Commit Branch: HEAD
UTC Build Time: 2022-05-19 04:37:08
Rust Version: rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Enable Features: jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile: dist_release
2022-05-19T18:30:12.351+0800 INFO k8s/client.go:107 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
./pd-server -V
Release Version: v6.1.0-nightly
Edition: Community
Git Commit Hash: 85918318d7a127a845af25e208b82b153820fc0d
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-05-18 10:57:30
2022-05-19T18:30:12.527+0800 INFO k8s/client.go:107 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
./tidb-server -V
Release Version: v6.1.0-nightly
Edition: Community
Git Commit Hash: 381e870c5ca018d9b5b6e89233d2cbd9a3b41cac
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-05-18 11:04:58
GoVersion: go1.18.2
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

What operating system and CPU are you using?

8core、16GB

Steps to reproduce

update config repeatedly while run tpcc
[2022/05/19 18:42:08.656 +08:00] [INFO] [cmd.go:124] ["Start remote command"] [cmd="go-tpc tpcc run -D tpcc --host tc-tidb.glh-dynamic-regions-jw8pv -P4000 --warehouses 1000 -T 64 --time 36000m --ignore-error '2013,1213,1105,1205,8022,8027,8028,9004,9007,9001'"] [nodename=benchtoolset]

What did you expect?

big region can split

What did happened?

current config
"coprocessor": {
"split-region-on-table": false,
"batch-split-limit": 10,
"region-max-size": "800MiB",
"region-split-size": "512MiB",
"region-max-keys": 8000000,
"region-split-keys": 4800000,
"consistency-check-method": "mvcc",
"enable-region-bucket": true,
"region-bucket-size": "128MiB",
"region-size-threshold-for-approximate": "384MiB",
"region-bucket-merge-size-ratio": 0.33
},

"max-merge-region-keys": 1000000,
"max-merge-region-size": 100,

Some large regions did not split as expected
eGUffDgG8s
/pd-ctl region topsize 1
image
@Lily2025
Copy link
Author

/type bug
/severity major
/assign 5kbpers

@tonyxuqqi tonyxuqqi mentioned this issue May 23, 2022
Merged
@bufferflies bufferflies mentioned this issue May 24, 2022
Closed
@Lily2025
Copy link
Author

/remove-label may-affects-4.0
/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2
/remove-label may-affects-5.3
/remove-label may-affects-5.4
/remove-label may-affects-6.0
/label affects-6.1

@nolouch
Copy link
Contributor

nolouch commented May 25, 2022

duplicated with #12597.

@nolouch nolouch closed this as completed May 25, 2022
@nolouch nolouch reopened this May 25, 2022
@nolouch nolouch mentioned this issue May 25, 2022
Closed
ti-chi-bot pushed a commit that referenced this issue May 27, 2022
@tonyxuqqi
raftstore: bucket and region split bug fix (#12620)
93fecdc 
close #12597

1) originally the split by keys only support CheckPolicy::Scan. Add CheckPolicy::Approximate support. 
2) refine the way to use approximate size for split---calculate the actual split count per region size instead of use batch_split_limit
3) refine the bucket_version logic to make it monotonic----basically keep the region_buckets in last_region_buckets on split/merge

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
@ti-srebot ti-srebot mentioned this issue May 27, 2022
Merged
@tonyxuqqi tonyxuqqi mentioned this issue May 27, 2022
Merged
ti-chi-bot added a commit that referenced this issue May 31, 2022
@ti-srebot @tonyxuqqi @ti-chi-bot
raftstore: bucket and region split bug fix (#12620) (#12674)
c1d2fe7 
close #12597, ref #12620

1) originally the split by keys only support CheckPolicy::Scan. Add CheckPolicy::Approximate support. 
2) refine the way to use approximate size for split---calculate the actual split count per region size instead of use batch_split_limit
3) refine the bucket_version logic to make it monotonic----basically keep the region_buckets in last_region_buckets on split/merge

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ti-chi-bot added a commit that referenced this issue Jun 1, 2022
@tonyxuqqi @ti-chi-bot
raftstore: use approximate size to generate bucket unless split regio…
f7edbcf 
…n check is needed and scan is used for it. (#12678)

ref #12597

When bucket is enabled,  using CheckPolicy::Scan would lead to large amount of read IO after tikv restart. Before this PR, the Scan is used unless the region size reaches 1.5G, which is very rare for a 96 ~ 256MB's region-split-size. After this change, generating bucket won't introduce new scan unless the scan is necessary for splitting region. This can significantly reduce the read IO.
Also refine the logic for the fix of 12597.

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
@ti-srebot ti-srebot mentioned this issue Jun 1, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Jun 1, 2022
@ti-srebot @tonyxuqqi
raftstore: use approximate size to generate bucket unless split regio…
50109b2 
…n check is needed and scan is used for it. (#12678) (#12714)

ref #12597, ref #12678

When bucket is enabled,  using CheckPolicy::Scan would lead to large amount of read IO after tikv restart. Before this PR, the Scan is used unless the region size reaches 1.5G, which is very rare for a 96 ~ 256MB's region-split-size. After this change, generating bucket won't introduce new scan unless the scan is necessary for splitting region. This can significantly reduce the read IO.
Also refine the logic for the fix of 12597.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
joccau pushed a commit to joccau/tikv that referenced this issue Jun 23, 2022
@ti-srebot @tonyxuqqi
@ti-chi-bot @joccau
raftstore: bucket and region split bug fix (tikv#12620) (tikv#12674)
0e07436 
close tikv#12597, ref tikv#12620

1) originally the split by keys only support CheckPolicy::Scan. Add CheckPolicy::Approximate support.
2) refine the way to use approximate size for split---calculate the actual split count per region size instead of use batch_split_limit
3) refine the bucket_version logic to make it monotonic----basically keep the region_buckets in last_region_buckets on split/merge

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: joccau <zak.zhao@pingcap.com>
joccau pushed a commit to joccau/tikv that referenced this issue Jun 23, 2022
@ti-srebot @tonyxuqqi @joccau
raftstore: use approximate size to generate bucket unless split regio…
077bacf 
…n check is needed and scan is used for it. (tikv#12678) (tikv#12714)

ref tikv#12597, ref tikv#12678

When bucket is enabled,  using CheckPolicy::Scan would lead to large amount of read IO after tikv restart. Before this PR, the Scan is used unless the region size reaches 1.5G, which is very rare for a 96 ~ 256MB's region-split-size. After this change, generating bucket won't introduce new scan unless the scan is necessary for splitting region. This can significantly reduce the read IO.
Also refine the logic for the fix of 12597.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
Signed-off-by: joccau <zak.zhao@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@5kbpers 5kbpers

Labels
affects-6.1 severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

raftstore: bucket and region split bug fix tonyxuqqi/tikv
5 participants
@jebter @nolouch @5kbpers @ti-chi-bot @Lily2025