-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(storage/dataflux): run worksteal listing parallel to sequential listing #10966
base: main
Are you sure you want to change the base?
Conversation
} | ||
|
||
// nextPage lists objects using the given lister options. | ||
func nextPage(ctx context.Context, opts nextPageOpts) (*nextPageResult, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method seems a little convoluted; do you think there is a way to simplify it or make it a little easier to read?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update nextPage. Let me know if its more readable now.
feat: run worksteal listing parallel to sequential listing
Dataflux fast-listing leverages worksteal algorithm to quickly list objects in a bucket by running several parallel processes. Worksteal algorithm splits a given namespace into multiple ranges for multiple workers(goroutines) to list objects in gcs bucket in parallel.
Adding logic to list next page in worksteal listing and basic unit tests that uses storage/emulator.
Fixes #10731