-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion #56158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion #56158
Conversation
…of wait_for_completion
shahar1
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add unit tests to ensure that the deprecation works properly?
|
sure @shahar1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR deprecates the wait_policy parameter in EmrCreateJobFlowOperator and replaces it with wait_for_completion to maintain consistency with other Amazon operators while preserving backward compatibility.
- Deprecated
wait_policyparameter with a warning message - Added
wait_for_completionboolean parameter as the new standard - Updated implementation to use
wait_for_completioninternally with backward compatibility mapping
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py | Updates operator constructor and execute method to use wait_for_completion instead of wait_policy |
| providers/amazon/tests/unit/amazon/aws/operators/test_emr_create_job_flow.py | Updates test cases to use new parameter and adds deprecation warning test |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py
Outdated
Show resolved
Hide resolved
|
@laksh-krishna-sharma could you please review Copilot's comments and check if they make sense? |
|
yes @shahar1 i am reviewing the comments |
|
@shahar1 the copilot comments make sense. I’m working on resolving these. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py
Outdated
Show resolved
Hide resolved
|
@shahar1 Can I mark the above Copilot comments as resolved? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py
Outdated
Show resolved
Hide resolved
|
@laksh-krishna-sharma please tag me when you're done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py
Outdated
Show resolved
Hide resolved
|
@shahar1 please review the changes and let me know if i have to refactor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
@shahar1 please review the changes and let me know if i have to refactor |
vincbeck
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One nit but overall looks good!
providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py
Outdated
Show resolved
Hide resolved
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
…of wait_for_completion (apache#56158) * refactor: deprecate wait_policy in EmrCreateJobFlowOperator in favor of wait_for_completion * added unit test for refactor * resolved copilot comments * resolved copilot comments * fixed failing test * fixed: refactor of wait_policy * ensured backward compatibility * removed "self.wait_policy = wait_policy"
Summary of changes
Closes #56153
wait_policyparameter inEmrCreateJobFlowOperator.wait_for_completion(boolean) as the standard parameter (consistent with other Amazon operators).wait_policy→wait_for_completion.Changes made
wait_for_completion.wait_policywith warning.