Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support directly sharded shuffles #95

Open
jonhoo opened this issue Sep 19, 2018 · 0 comments
Open

Support directly sharded shuffles #95

jonhoo opened this issue Sep 19, 2018 · 0 comments
Labels
enhancement New feature or request m-distribution Related to distributed operation m-operators Related to operator implementations and semantics m-partial Related to partial data-flow, ancestor queries, and backfills m-sharding Related to data-flow sharding perf Performance-oriented issues research This issue represents an open research question, and will require careful thought to fix.

Comments

@jonhoo
Copy link
Contributor

jonhoo commented Sep 19, 2018

If a shuffle is needed, we currently inject a "merge-then-shard" sequence. This produces correct results, but also creates an artificial bottleneck. In theory, the leaves in the shards of the source domain should be able to directly send their output to the appropriate shards instead. Where this gets tricky is with ancestor queries. If an ancestor query has to ask multiple shards of the source domain, there is no longer a union in place that can buffer and merge the resulting backfills correctly.

@jonhoo jonhoo added enhancement New feature or request m-partial Related to partial data-flow, ancestor queries, and backfills perf Performance-oriented issues m-operators Related to operator implementations and semantics m-sharding Related to data-flow sharding m-distribution Related to distributed operation research This issue represents an open research question, and will require careful thought to fix. labels Sep 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request m-distribution Related to distributed operation m-operators Related to operator implementations and semantics m-partial Related to partial data-flow, ancestor queries, and backfills m-sharding Related to data-flow sharding perf Performance-oriented issues research This issue represents an open research question, and will require careful thought to fix.
Projects
None yet
Development

No branches or pull requests

1 participant