Improve exception handling in AzureDataFactoryTrigger #56350
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling.
Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened.
When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt.
boring-cyborg
bot
added
area:providers
provider:microsoft-azure
prdai
reviewed
Oct 3, 2025
providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py
Show resolved
Hide resolved
ashb
reviewed
Oct 3, 2025
providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py
Outdated
Show resolved
Hide resolved
…e/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
dstandish
approved these changes
Oct 13, 2025
potiuk
approved these changes
Oct 13, 2025
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 14, 2025
* fix: don't wrap exception in AirflowException Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling. * fix: do log the actual exception in the trigger Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened. * fix: don't override original exception when cancelling ADF pipeline run When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt. * Update providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: remove unused exception --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
dstandish
changed the title
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 15, 2025
* fix: don't wrap exception in AirflowException Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling. * fix: do log the actual exception in the trigger Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened. * fix: don't override original exception when cancelling ADF pipeline run When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt. * Update providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: remove unused exception --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 17, 2025
* fix: don't wrap exception in AirflowException Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling. * fix: do log the actual exception in the trigger Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened. * fix: don't override original exception when cancelling ADF pipeline run When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt. * Update providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: remove unused exception --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 19, 2025
* fix: don't wrap exception in AirflowException Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling. * fix: do log the actual exception in the trigger Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened. * fix: don't override original exception when cancelling ADF pipeline run When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt. * Update providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: remove unused exception --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
69 tasks
TyrellHaywood
pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
* fix: don't wrap exception in AirflowException Wrapping the exception AirflowException serves no real purpose and obscures the original exception for downstream handling. * fix: do log the actual exception in the trigger Although the exception text is passed back to the operator, we lose the stack trace while doing so. By logging the exception in the trigger first, we will get full details about where the exception happened. * fix: don't override original exception when cancelling ADF pipeline run When we fail to cancel a pipeline run after an unexpected error, we are overriding the actual exception which caused the failure in first place. This change addresses this by keeping the original exception untouched and instead log any exception that occurs during the cancellation attempt. * Update providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: remove unused exception --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are instances when
AzureDataFactoryOperatorin deferrable mode fails with the following error:Unfortunately this error is only a result of another error resulting in an attempt to cancel an existing pipeline run. The shown error message indicates that this attempt failed, because the run itself was already successful. Unfortunately that second error is the only error reported back to the operator even though it is not the initial cause of failure (which is unknown).
This PR addresses this by:
Additionally we do not obfuscate the exception when cancelling fails, such that we get better visibility on the actual exception thrown, which may allow us to handle failed cancellations for already successful pipelines more gracefully in the future.