-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add wait_policy option to EmrCreateJobFlowOperator. #44055
Add wait_policy option to EmrCreateJobFlowOperator. #44055
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
020522f
to
3329599
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you cover this new use case in a unit test?
3329599
to
30c0240
Compare
30c0240
to
fc111c8
Compare
5212fa9
to
d5f85bc
Compare
Good job! I like that version |
48c9119
to
68f7950
Compare
9d81e5e
to
25dadd3
Compare
f447927
to
62e4f1e
Compare
I'm not sure what's the next step here but from my side everything is ready. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks good! Just one nit and you do not a newsfragment for this change
b59a5f8
to
c425121
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perfect!
c425121
to
f971512
Compare
Should be ready for merge |
Possible values: - None: No wait (default) - WaitPolicy.WAIT_FOR_COMPLETION: Previous behaviour when wait_for_completion was True - WaitPolicy.WAIT_FOR_STEPS_COMPLETION: New behaviour - wait for the cluster to terminate.
f971512
to
de09376
Compare
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Currently, when using
wait_for_complition=True
it will trigger the node to only wait until the cluster is successfully launched.I added the option
wait_policy
so that one can chose what waiter will be used.Possible values:
None
: No wait (default)WaitPolicy.WAIT_FOR_COMPLETION
: Previous behaviour whenwait_for_completion
wasTrue
WaitPolicy.WAIT_FOR_STEPS_COMPLETION
: New behaviour - wait for the cluster to terminate.This is useful because it will eliminate the need to have an extra node in your DAG.