Add cancel_on_kill support for EMR Serverless deferrable operator#60440
Merged
BasPH merged 16 commits intoapache:mainfrom Feb 5, 2026
Merged
Add cancel_on_kill support for EMR Serverless deferrable operator#60440BasPH merged 16 commits intoapache:mainfrom
BasPH merged 16 commits intoapache:mainfrom
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
BasPH
reviewed
Jan 15, 2026
Contributor
BasPH
left a comment
There was a problem hiding this comment.
first quick scan, few comments
o-nikolas
reviewed
Jan 16, 2026
BasPH
approved these changes
Feb 4, 2026
Contributor
|
@cruseakshay could you fix the failing test? |
Apply prek hook formatting to single-line assertion. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace deprecated session.query().filter() with select().where() and session.scalars() pattern. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Done |
Alok-kumar-priyadarshi
pushed a commit
to Alok-kumar-priyadarshi/airflow
that referenced
this pull request
Feb 5, 2026
…ache#60440) * Add cancel_on_kill support for EMR Serverless deferrable operator * Add details to safe_to_cancel method. * Fix formatting in EMR trigger test Apply prek hook formatting to single-line assertion. Co-authored-by: Cursor <cursoragent@cursor.com> * Use modern SQLAlchemy 2.0 style query Replace deprecated session.query().filter() with select().where() and session.scalars() pattern. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Akshay <akshaykumar.salunke@astronomer.io> Co-authored-by: Akshay <cruseakshay@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
jhgoebbert
pushed a commit
to jhgoebbert/airflow_Owen-CH-Leung
that referenced
this pull request
Feb 8, 2026
…ache#60440) * Add cancel_on_kill support for EMR Serverless deferrable operator * Add details to safe_to_cancel method. * Fix formatting in EMR trigger test Apply prek hook formatting to single-line assertion. Co-authored-by: Cursor <cursoragent@cursor.com> * Use modern SQLAlchemy 2.0 style query Replace deprecated session.query().filter() with select().where() and session.scalars() pattern. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Akshay <akshaykumar.salunke@astronomer.io> Co-authored-by: Akshay <cruseakshay@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
81 tasks
Ratasa143
pushed a commit
to Ratasa143/airflow
that referenced
this pull request
Feb 15, 2026
…ache#60440) * Add cancel_on_kill support for EMR Serverless deferrable operator * Add details to safe_to_cancel method. * Fix formatting in EMR trigger test Apply prek hook formatting to single-line assertion. Co-authored-by: Cursor <cursoragent@cursor.com> * Use modern SQLAlchemy 2.0 style query Replace deprecated session.query().filter() with select().where() and session.scalars() pattern. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Akshay <akshaykumar.salunke@astronomer.io> Co-authored-by: Akshay <cruseakshay@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for cancelling EMR Serverless jobs when an Airflow task is killed while in deferrable mode.
Problem:
When using
EmrServerlessStartJobOperatorwithdeferrable=True, if a user cancels the task in Airflow, the EMR Serverless job continues running in AWS. This leads to orphaned jobs consuming resourcesSolution:
Implements the "cancel_on_kill" (similar to
DataprocSubmitJobTrigger) for EMR Serverless:cancel_on_killparameter toEmrServerlessStartJobOperator(default:True)cancel_on_killparameter toEmrServerlessStartJobTriggersafe_to_cancel()method in the trigger to distinguish between:run()method to catchasyncio.CancelledErrorand cancel the EMR job when safeChanges:
providers/amazon/aws/operators/emr.py: Addedcancel_on_killparameterproviders/amazon/aws/triggers/emr.py: Added cancellation logic withsafe_to_cancel()tests/unit/amazon/aws/triggers/test_emr.py: Added unit tests for cancellation scenariosTesting:
Was generative AI tooling used to co-author this PR?
Generated-by: Claude (Cursor IDE) following the guidelines
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.