Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure #55728
Description
Apache Airflow Provider(s)
microsoft-azure
Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==12.6.1
Apache Airflow version
3.0.6
Operating System
Linux
Deployment
Astronomer
Deployment details
No response
What happened
Running the AzureDataFactoryRunPipelineOperator async + Airflow 3.0.3 gives this failure:
ERROR - Task failed with exception: source="task"
AirflowException: You cannot use AsyncToSync in the same thread as an async event loop - just await the async function directly.
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 920 in run
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1215 in _execute_task
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 1606 in resume_execution
File "/usr/local/lib/python3.12/site-packages/airflow/providers/microsoft/azure/operators/data_factory.py", line 256 in execute_complete
The operator runs fine on Airflow 2.
What you think should happen instead
The operator should not fail
How to reproduce
Run AzureDataFactoryRunPipelineOperator async + Airflow 3.0.3 or higher
Anything else
Same error also found in other issues/PRs:
- MSGraphAsyncOperator fails to call API #54350
- RuntimeError in trigger creating hooks, Airflow 3.0.3 #53447
- Fix sync to async issue when getting connection for SQLExecuteQueryTrigger #55094
- Avoid using extra_dejson method on connection in KiotaRequestAdapterHook to avoid AsyncToSync RuntimeError #55179
- Made get_conn method async for KiotaRequestAdapterHook to fix AsyncToSync error #54598
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct