Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,9 @@ class EmrAddStepsOperator(AwsBaseOperator[EmrHook]):
:param steps: boto3 style steps or reference to a steps file (must be '.json') to
be added to the jobflow. (templated)
:param wait_for_completion: If True, the operator will wait for all the steps to be completed.
Defaults to False. Note: When deferrable=True, this parameter will not take effect.
:param execution_role_arn: The ARN of the runtime role for a step on the cluster.
:param do_xcom_push: if True, job_flow_id is pushed to XCom with key job_flow_id.
:param wait_for_completion: Whether to wait for job run completion. (default: True)
:param deferrable: If True, the operator will wait asynchronously for the job to complete.
This implies waiting for completion. This mode requires aiobotocore module to be installed.
(default: False)
Expand Down Expand Up @@ -970,7 +970,7 @@ class EmrServerlessCreateApplicationOperator(AwsBaseOperator[EmrServerlessHook])

:param release_label: The EMR release version associated with the application.
:param job_type: The type of application you want to start, such as Spark or Hive.
:param wait_for_completion: If true, wait for the Application to start before returning. Default to True.
:param wait_for_completion: If true, wait for the Application to start before returning. Defaults to True.
If set to False, ``waiter_max_attempts`` and ``waiter_delay`` will only be applied when
waiting for the application to be in the ``CREATED`` state.
:param client_request_token: The client idempotency token of the application to create.
Expand All @@ -985,8 +985,9 @@ class EmrServerlessCreateApplicationOperator(AwsBaseOperator[EmrServerlessHook])
:param verify: Whether or not to verify SSL certificates. See:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
:param waiter_max_attempts: Number of times the waiter should poll the application to check the state.
If not set, the waiter will use its default value.
Defaults to 25 if not set.
:param waiter_delay: Number of seconds between polling the state of the application.
Defaults to 60 seconds if not set.
:param deferrable: If True, the operator will wait asynchronously for application to be created.
This implies waiting for completion. This mode requires aiobotocore module to be installed.
(default: False, but can be overridden in config file by setting default_deferrable to True)
Expand Down Expand Up @@ -1117,8 +1118,8 @@ class EmrServerlessStartJobOperator(AwsBaseOperator[EmrServerlessHook]):
Its value must be unique for each request.
:param config: Optional dictionary for arbitrary parameters to the boto API start_job_run call.
:param wait_for_completion: If true, waits for the job to start before returning. Defaults to True.
If set to False, ``waiter_countdown`` and ``waiter_check_interval_seconds`` will only be applied
when waiting for the application be to in the ``STARTED`` state.
If set to False, ``waiter_max_attempts`` and ``waiter_delay`` will only be applied
when waiting for the application to be in the ``STARTED`` state.
:param aws_conn_id: The Airflow connection used for AWS credentials.
If this is ``None`` or empty then the default boto3 behaviour is used. If
running Airflow in a distributed manner and aws_conn_id is None or
Expand All @@ -1129,8 +1130,9 @@ class EmrServerlessStartJobOperator(AwsBaseOperator[EmrServerlessHook]):
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
:param name: Name for the EMR Serverless job. If not provided, a default name will be assigned.
:param waiter_max_attempts: Number of times the waiter should poll the application to check the state.
If not set, the waiter will use its default value.
Defaults to 25 if not set.
:param waiter_delay: Number of seconds between polling the state of the job run.
Defaults to 60 seconds if not set.
:param deferrable: If True, the operator will wait asynchronously for the crawl to complete.
This implies waiting for completion. This mode requires aiobotocore module to be installed.
(default: False, but can be overridden in config file by setting default_deferrable to True)
Expand Down Expand Up @@ -1439,9 +1441,9 @@ class EmrServerlessStopApplicationOperator(AwsBaseOperator[EmrServerlessHook]):
If you want to wait for the jobs to finish gracefully, use
:class:`airflow.providers.amazon.aws.sensors.emr.EmrServerlessJobSensor`
:param waiter_max_attempts: Number of times the waiter should poll the application to check the state.
Default is 25.
Defaults to 25 if not set.
:param waiter_delay: Number of seconds between polling the state of the application.
Default is 60 seconds.
Defaults to 60 seconds if not set.
:param deferrable: If True, the operator will wait asynchronously for the application to stop.
This implies waiting for completion. This mode requires aiobotocore module to be installed.
(default: False, but can be overridden in config file by setting default_deferrable to True)
Expand Down Expand Up @@ -1574,9 +1576,9 @@ class EmrServerlessDeleteApplicationOperator(EmrServerlessStopApplicationOperato
:param verify: Whether or not to verify SSL certificates. See:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
:param waiter_max_attempts: Number of times the waiter should poll the application to check the state.
Defaults to 25.
Defaults to 25 if not set.
:param waiter_delay: Number of seconds between polling the state of the application.
Defaults to 60 seconds.
Defaults to 60 seconds if not set.
:param deferrable: If True, the operator will wait asynchronously for application to be deleted.
This implies waiting for completion. This mode requires aiobotocore module to be installed.
(default: False, but can be overridden in config file by setting default_deferrable to True)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,15 +134,15 @@
force_stop=True,
)
# [END howto_operator_emr_serverless_stop_application]
stop_app.waiter_check_interval_seconds = 1
stop_app.waiter_delay = 1

# [START howto_operator_emr_serverless_delete_application]
delete_app = EmrServerlessDeleteApplicationOperator(
task_id="delete_application",
application_id=emr_serverless_app_id,
)
# [END howto_operator_emr_serverless_delete_application]
delete_app.waiter_check_interval_seconds = 1
delete_app.waiter_delay = 1
delete_app.trigger_rule = TriggerRule.ALL_DONE

delete_s3_bucket = S3DeleteBucketOperator(
Expand Down