-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate Ballista scheduler with DataFusion #64
Labels
enhancement
New feature or request
Comments
This was referenced Aug 8, 2021
tustvold
added a commit
that referenced
this issue
Jul 17, 2022
* Add optional serde support to datafusion-proto (#2889) * Add public methods for JSON serde (#64) * Misc suggestions * Update datafusion/proto/Cargo.toml Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> * Fixes * Fixup Cargo.toml * Format Cargo.toml Co-authored-by: Brent Gardner <bgardner@squarelabs.net>
7 tasks
I am not sure if this is tracking anything actionable and hasn't had any updates in a few years, so closing it. Please feel free to reopen if i got that wrong |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The Ballista scheduler breaks a query down into stages based on changes in partitioning in the plan, where each stage is broken down into tasks that can be executed concurrently.
Rather than trying to run all the partitions at once, Ballista executors process n concurrent tasks at a time and then request new tasks from the scheduler.
This approach would help DataFusion scale better and it would be ideal to use the same scheduler to scale across cores in DataFusion and across nodes in Ballista.
Describe the solution you'd like
Implement an extensible scheduler in DataFusion and have Ballista extend it to provide distributed execution.
Describe alternatives you've considered
None
Additional context
None
The text was updated successfully, but these errors were encountered: