Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: keyspace management #5265

Merged
merged 21 commits into from
Aug 8, 2022
Merged

storage: keyspace management #5265

merged 21 commits into from
Aug 8, 2022

Conversation

AmoebaProtozoa
Copy link
Contributor

@AmoebaProtozoa AmoebaProtozoa commented Jul 5, 2022 •
edited
Loading

Signed-off-by: David 8039876+AmoebaProtozoa@users.noreply.github.com

What problem does this PR solve?

Added storage methods for saving and loading keyspace metadata
Issue Number: ref #5293

What is changed and how does it work?

storage: keyspace management

Check List

Tests

  • Unit test

Code changes

Release note

None

@AmoebaProtozoa
storage: keyspace storage
54a9a53 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
storage: encode spaceID string as base 10
235d463 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 5, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • AndreMouche
  • nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Jul 5, 2022
@AmoebaProtozoa
storage: add save/load keyspace ID
1de1307 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa AmoebaProtozoa mentioned this pull request Jul 11, 2022
Closed
11 tasks
@AmoebaProtozoa
storage: add comments to keyspaceIDPath
a22119f 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
storage: remove keyspace
923454b 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
address comments
0c66986 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
address comments
8086113 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
add keyspaceID padding in etcd path
312a6d5 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
typo fix and kvproto update
8e6679f 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
Merge branch 'master' into keyspace_storage
8d43ab6
@AmoebaProtozoa
Merge branch 'master' of github.com:tikv/pd into keyspace_storage
3b76bd0
@AmoebaProtozoa
update go mod
c1ab719 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
storage: fix lint
f097c9b 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa AmoebaProtozoa changed the title [WIP] storage: keyspace management storage: keyspace management Jul 27, 2022
@AmoebaProtozoa AmoebaProtozoa marked this pull request as ready for review July 27, 2022 06:04
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 27, 2022
This was referenced Jul 27, 2022
Merged
Merged
AndreMouche
AndreMouche approved these changes Aug 3, 2022
View reviewed changes
Copy link
Member

@AndreMouche AndreMouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

server/storage/endpoint/key_path.go Show resolved Hide resolved
server/storage/endpoint/keyspace.go
// SaveKeyspace saves the given keyspace to the storage.
func (se *StorageEndpoint) SaveKeyspace(keyspace *keyspacepb.KeyspaceMeta) error {
key := KeyspaceMetaPath(keyspace.GetId())
return se.saveProto(key, keyspace)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we save json instead to make it read/debug-friendly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here some information defined in proto. maybe proto more suitable.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 3, 2022
@AmoebaProtozoa
update keyspace
52e3fb5 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
nolouch
nolouch reviewed Aug 5, 2022
View reviewed changes
go.mod Outdated Show resolved Hide resolved
@AmoebaProtozoa
update go mod
bca8542 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
Merge branch 'master' of github.com:tikv/pd into keyspace_storage
014f7d4
@codecov
Copy link

codecov bot commented Aug 7, 2022 •
edited
Loading

Codecov Report

Merging #5265 (67dc293) into master (99e38eb) will increase coverage by 0.07%.
The diff coverage is 79.45%.

@@            Coverage Diff             @@
##           master    #5265      +/-   ##
==========================================
+ Coverage   75.58%   75.66%   +0.07%     
==========================================
  Files         312      313       +1     
  Lines       31061    31104      +43     
==========================================
+ Hits        23479    23536      +57     
+ Misses       5572     5556      -16     
- Partials     2010     2012       +2
Flag Coverage Δ
unittests 75.66% <79.45%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
server/grpc_service.go 48.81% <ø> (-0.60%) ⬇️
server/storage/storage.go 71.15% <ø> (ø)
server/tso/global_allocator.go 61.25% <ø> (ø)
server/storage/endpoint/keyspace.go 73.33% <73.33%> (ø)
server/schedule/filter/filters.go 91.64% <79.41%> (-0.43%) ⬇️
server/storage/endpoint/key_path.go 100.00% <100.00%> (ø)
server/schedulers/shuffle_hot_region.go 55.55% <0.00%> (-10.11%) ⬇️
server/storage/kv/etcd_kv.go 79.71% <0.00%> (-5.80%) ⬇️
server/storage/endpoint/meta.go 63.63% <0.00%> (-2.28%) ⬇️
server/region_syncer/server.go 86.26% <0.00%> (-1.10%) ⬇️
... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@AmoebaProtozoa
update kvproto reference
eb4f3c5 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
rleungx
rleungx reviewed Aug 8, 2022
View reviewed changes
Copy link
Member

@rleungx rleungx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest LGTM.

server/storage/endpoint/key_path.go Outdated Show resolved Hide resolved
@AmoebaProtozoa
update comments
ce432f2 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@nolouch
Copy link
Contributor

nolouch commented Aug 8, 2022

ci failed. maybe you need run make tidy.

@AmoebaProtozoa
make tidy
8612c9c 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
@AmoebaProtozoa
update pd-tso-bench go sum
67dc293 
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
nolouch
nolouch approved these changes Aug 8, 2022
View reviewed changes
Copy link
Contributor

@nolouch nolouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 8, 2022
@nolouch
Copy link
Contributor

nolouch commented Aug 8, 2022

/merge

@ti-chi-bot
Copy link
Member

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 67dc293

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 8, 2022
@ti-chi-bot ti-chi-bot merged commit e089172 into tikv:master Aug 8, 2022
@AmoebaProtozoa AmoebaProtozoa deleted the keyspace_storage branch August 8, 2022 07:18
ti-chi-bot added a commit that referenced this pull request Feb 2, 2023
@AmoebaProtozoa @ti-chi-bot
apiv2: keyspace management (#5313)
abb62d2 
ref #5265, ref #5291, ref #5293, ref #5297

Add APIv2 HTTP interface for keyspace management.

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rleungx rleungx rleungx left review comments

@disksing disksing disksing left review comments

@sunxiaoguang sunxiaoguang sunxiaoguang left review comments

@ystaticy ystaticy ystaticy left review comments

@JmPotato JmPotato JmPotato left review comments

@nolouch nolouch nolouch approved these changes

@AndreMouche AndreMouche AndreMouche approved these changes

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@AmoebaProtozoa @ti-chi-bot @nolouch @AndreMouche @JmPotato @ystaticy @sunxiaoguang @disksing @rleungx