Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

region is not balanced during br restore #7148

Closed
Lily2025 opened this issue Sep 25, 2023 · 6 comments · Fixed by #7149
Closed

region is not balanced during br restore #7148

Lily2025 opened this issue Sep 25, 2023 · 6 comments · Fixed by #7149
Assignees
@CabinfeverB
Labels
affects-6.5 severity/major type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Bug Report

What did you do?

1、command="export AWS_ACCESS_KEY_ID=minioadmin;export AWS_SECRET_ACCESS_KEY=minioadmin;/br restore db --send-credentials-to-tikv=true --db tpcc20000 --pd http://tc-pd.endless-ha-test-with-some-feature-tps-2760236-1-101:2379 --storage s3://benchmark/tpcc20000 --s3.endpoint http://minio.pingcap.net:9000 --check-requirements=false

What did you expect to see?

region is balanced during br restore

What did you see instead?

region is not balanced during br restore
697e5e61-9453-4c4b-93b1-176e7eec52e5

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v7.4.0-alpha
Edition: Community
Git Commit Hash: a21fd58
Git Branch: heads/refs/tags/v7.4.0-alpha
UTC Build Time: 2023-09-22 11:36:18
2023-09-23T01:25:02.465+0800
@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Sep 25, 2023
@Lily2025
Copy link
Author

/severity major

@Lily2025
Copy link
Author

/assign rleungx

@rleungx rleungx assigned CabinfeverB and unassigned rleungx Sep 25, 2023
@CabinfeverB CabinfeverB mentioned this issue Sep 25, 2023
Merged
@CabinfeverB
Copy link
Member

affects 6.5.5

@CabinfeverB CabinfeverB mentioned this issue Sep 25, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 25, 2023
@CabinfeverB
Revert "statistics: fix empty region count when resuming (#7009)" (#7149
eac55a7 
)

close #7148

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
@ti-chi-bot ti-chi-bot mentioned this issue Sep 25, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Sep 25, 2023
@CabinfeverB @ti-chi-bot
This is an automated cherry-pick of tikv#7149
dd6d08d 
close tikv#7148

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this issue Sep 26, 2023
@CabinfeverB
Revert "statistics: fix empty region count when resuming (#7009) (#70… (
b0207a6 
#7150)

ref #7148

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
This was referenced Sep 26, 2023
Closed
Closed
@CabinfeverB
Copy link
Member

The BR suspends the merge scheduling operation by setting max-merge-region-size to 0.
Due to PR #7009, the size of the region where the spilt occurs is 0.
So the BR failed to suspend the merge scheduling operation. After the split interval, regions are merged, which weakens scatter effect and results in unbalanced TiKV data (region score).

@CabinfeverB
Copy link
Member

Workaround
Pause the schedule manually with the limit parameter. In pd-ctl:

config set schedule.merge-schedule-limit 0

@CabinfeverB
Copy link
Member

There may be a bug about size check after split, see tikv/tikv#15696

@3pointer 3pointer mentioned this issue Oct 20, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit to pingcap/tidb that referenced this issue Nov 2, 2023
@3pointer
br: use merge-schedule-limit to pause merge (#47576)
341aebe 
ref tikv/pd#7148
This was referenced Nov 2, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit to pingcap/tidb that referenced this issue Nov 22, 2023
@ti-chi-bot
br: use merge-schedule-limit to pause merge (#47576) (#48247)
17f58e8 
ref tikv/pd#7148
@niubell niubell mentioned this issue Nov 30, 2023
Merged
16 tasks
ti-chi-bot bot pushed a commit to pingcap/tidb that referenced this issue Feb 21, 2024
@ti-chi-bot
br: use merge-schedule-limit to pause merge (#47576) (#48246)
a984f98 
ref tikv/pd#7148
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 21, 2024
@CabinfeverB @ti-chi-bot
This is an automated cherry-pick of tikv#7149
98b978a 
close tikv#7148

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 21, 2024
Closed
ti-chi-bot bot pushed a commit to pingcap/tidb that referenced this issue Nov 6, 2024
@ti-chi-bot
br: use merge-schedule-limit to pause merge (#47576) (#48248)
a92e555 
ref tikv/pd#7148
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CabinfeverB CabinfeverB

Labels
affects-6.5 severity/major type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "statistics: fix empty region count when resuming (#7009)" CabinfeverB/pd
4 participants
@CabinfeverB @rleungx @ti-chi-bot @Lily2025