Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We started to use
@AwsBaseHook.retry(should_retry)
on_start_task
of theECSOperator
in Airflow 2.1.0 (or providers-amazon >= 1.3.0).airflow/airflow/providers/amazon/aws/operators/ecs.py
Line 224 in 304e174
which is a little bit tricky according to AWS support:
response['failures']
is not always reliable, which means an ECS task may still be provisioned successfully even when we get "failures" in theresponse
. The only way to be sure is to useDescribeTasks
to see if the task is provisioned correctly.With the
@AwsBaseHook.retry(should_retry)
on_start_task()
, i started to see my Airflow sending multipleRunTask
events to AWS for the same task and I ended up with multiple instances running and they are interfering with each other since they require the same back-end resources. This also creates many "ghost" ECS tasks that are not under Airflow's radar and quite difficult to debug 😿I don't know if using the retry is a way of dealing with this #15000 issue.
But i'd rather go back to the old way at the moment.
Happy to discuss further on this.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.