Support directly sharded shuffles #95
Labels
enhancement
New feature or request
m-distribution
Related to distributed operation
m-operators
Related to operator implementations and semantics
m-partial
Related to partial data-flow, ancestor queries, and backfills
m-sharding
Related to data-flow sharding
perf
Performance-oriented issues
research
This issue represents an open research question, and will require careful thought to fix.
If a shuffle is needed, we currently inject a "merge-then-shard" sequence. This produces correct results, but also creates an artificial bottleneck. In theory, the leaves in the shards of the source domain should be able to directly send their output to the appropriate shards instead. Where this gets tricky is with ancestor queries. If an ancestor query has to ask multiple shards of the source domain, there is no longer a union in place that can buffer and merge the resulting backfills correctly.
The text was updated successfully, but these errors were encountered: