KV scan batch size control #4875
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nopcoder
force-pushed
the
feature/kv-batch-size
branch
from
c7faa13 to
fd09890
Compare
guy-har
requested changes
Dec 28, 2022
| @@ -60,6 +60,11 @@ type ValueWithPredicate struct { | |||
| Predicate Predicate | |||
| } | |||
|
|
|||
| type ScanOptions struct { | |||
| KeyStart []byte | |||
| BatchSize int | |||
There was a problem hiding this comment.
Maybe we should add some documentation here explaining what BatchSize represents
pkg/kv/store.go
Outdated
| @@ -79,7 +84,9 @@ type Store interface { | |||
|
|
|||
| // Scan returns entries that can be read by key order, starting at or after the `start` position | |||
There was a problem hiding this comment.
Suggested change
| // Scan returns entries that can be read by key order, starting at or after the `start` position | |
| // Scan returns entries that can be read by key order |
pkg/kv/store.go
Outdated
| @@ -79,7 +84,9 @@ type Store interface { | |||
|
|
|||
| // Scan returns entries that can be read by key order, starting at or after the `start` position | |||
| // partitionKey is optional, passing it might increase performance. | |||
| Scan(ctx context.Context, partitionKey, start []byte) (EntriesIterator, error) | |||
| // 'options' holds additional parameters to limit the number of records | |||
| // and set the prefix to scan. | |||
There was a problem hiding this comment.
does it set the prefix to scan or the start key?
There was a problem hiding this comment.
true, fixing the comment.
pkg/graveler/graveler.go
Outdated
| @@ -1872,7 +1873,7 @@ func (g *Graveler) addCommitNoLock(ctx context.Context, repository *RepositoryRe | |||
| } | |||
|
|
|||
| func (g *Graveler) isStagingEmpty(ctx context.Context, repository *RepositoryRecord, branch *Branch) (bool, error) { | |||
| itr, err := g.listStagingArea(ctx, branch) | |||
| itr, err := g.listStagingArea(ctx, branch, 0) | |||
There was a problem hiding this comment.
Suggested change
| itr, err := g.listStagingArea(ctx, branch, 0) | |
| itr, err := g.listStagingArea(ctx, branch, 1) |
There was a problem hiding this comment.
Thanks - good call to optimize empty checks too
pkg/graveler/graveler.go
Outdated
| @@ -1904,7 +1905,7 @@ func (g *Graveler) isSealedEmpty(ctx context.Context, repository *RepositoryReco | |||
| if len(branch.SealedTokens) == 0 { | |||
| return true, nil | |||
| } | |||
| itrs, err := g.sealedTokensIterator(ctx, branch) | |||
| itrs, err := g.sealedTokensIterator(ctx, branch, 0) | |||
There was a problem hiding this comment.
Suggested change
| itrs, err := g.sealedTokensIterator(ctx, branch, 0) | |
| itrs, err := g.sealedTokensIterator(ctx, branch, 1) |
There was a problem hiding this comment.
Thanks - good call to optimize empty checks too
pkg/graveler/staging/manager.go
Outdated
| @@ -94,8 +94,8 @@ func (m *Manager) DropKey(ctx context.Context, st graveler.StagingToken, key gra | |||
|
|
|||
| // List TODO niro: Remove batchSize parameter post KV | |||
There was a problem hiding this comment.
Think we can remove this todo
Comment on lines
+204
to
+206
| if opts.PrefetchSize > 0 { | ||
| opts.PrefetchValues = true | ||
| } |
pkg/kv/dynamodb/store.go
Outdated
Comment on lines
357
to
362
| if s.params.ScanLimit != 0 { | ||
| queryInput.SetLimit(s.params.ScanLimit) | ||
| } | ||
| if options.BatchSize != 0 { | ||
| queryInput.Limit = aws.Int64(int64(options.BatchSize)) | ||
| } |
There was a problem hiding this comment.
We should choose the minimum between ScanLimit and batchSize
nopcoder
added
the
include-changelog
guy-har
approved these changes
Dec 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Close #4712
Pass listing in catalog level to the KV driver level.
Listing works with paginations, while KV with iterators.
In this PR the caller can pass recommended batch size to the KV level so the implementation can utilize our resources better and perform faster for specific actions.