Add best-effort cleanup to EcsRunTaskOperator on post-start failure

[SameerMesiah97]

Description

Added best-effort cleanup to EcsRunTaskOperator to ensure ECS tasks are stopped when failures occur after a task has been successfully started. Cleanup behavior is guarded by a flag and is opted in by default.

Previously, the operator could successfully start an ECS task via RunTask and then fail during post-start steps (for example, when waiting for task completion with wait_for_completion=True and missing ecs:DescribeTasks permissions). In these cases, the Airflow task failed while the ECS task continued running in AWS.

The operator now attempts to stop any ECS task that was started by the current task instance if an exception is raised after task start. Cleanup is performed opportunistically and does not mask or replace the original exception if stopping the task fails. It is only triggered for post-start failures (throwingWaiterError), ensuring it runs only when an ECS task has been created and avoiding interception of non-AWS exceptions.

Rationale

EcsRunTaskOperator manages the lifecycle of an external resource whose execution extends beyond the lifetime of the Airflow task. If task start succeeds but subsequent execution steps fail, Airflow can no longer reliably observe or manage the running ECS task, potentially leaving resources running unexpectedly.

Failures after task start can occur for multiple reasons, including IAM permission errors (for example, missing ecs:DescribeTasks) or loss of access to systems used during task execution. Attempting best-effort cleanup in these scenarios avoids leaving unmanaged ECS tasks running while preserving existing failure semantics.

Cleanup is only attempted when the operator can confidently determine that the ECS task was started by the current execution. This is achieved by tracking whether the task was started during the current run and using the task ARN returned by RunTask. This avoids interfering with pre-existing tasks in reattach scenarios while still preventing resource leaks on post-start failures.

Restricting cleanup to WaiterError prevents unintended side effects from catching unrelated failures while still addressing orphaned ECS tasks created during execution.

Tests

• Added a unit test verifying that an ECS task is stopped when a failure occurs after task start.
• Added a unit test ensuring that failures during cleanup do not mask or override the original exception.

Documentation

The docstring for EcsRunTaskOperator has been updated with a brief description of the new flag stop_task_on_failure .

Backwards Compatibility

A new flag called stop_task_on_failure has been added to EcsRunTaskOperator' with a default setting of True. Cleanup will now be attempted on a best-effort basis if WaiterError` is encountered.

Closes: #61050