Enhance DataFusion Integration with Asynchronous and Deferrable Support #45850
Labels
area:async-operators
AIP-40: Deferrable ("Async") Operators
good first issue
kind:feature
Feature Requests
provider:google
Google (including GCP) related issues
Comments
jjavieralonso
added
kind:feature
dosubot
bot
added
area:async-operators
|
Sure if you want to submit it - feel free. Other than that it's just a (reasonable) wish that might be implemented if someone wants to tackle it. Marking as good first issue |
potiuk
added
good first issue
and removed
needs-triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
This feature enhances the DataFusion integration in Apache Airflow by adding asynchronous and deferrable support to DataFusion operators and hooks. Additionally, it introduces new DataFusion links and improves error handling.
Use case/motivation
The addition of asynchronous and deferrable support allows users to run DataFusion pipelines more efficiently by leveraging non-blocking I/O operations. This is particularly useful for long-running pipelines, reducing the need for active waiting and improving resource utilization. The new DataFusion links provide easier navigation to specific DataFusion resources within the Google Cloud Console.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: