-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge all ECS executor configs following recursive python dict update #37137
Merge all ECS executor configs following recursive python dict update #37137
Conversation
CC @syedahsn (since I can't seem to add you via the reviewers list) |
@mshober Can you have a look at this PR and see if it satisfies your use case? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Funny that that helper function already existed 😅
LGTM. One thing that occurred to me is whether we want to document this design choice. I think the unit test does a very good job of explaining the behaviour. Might be worth putting that into the documentation.
I put a new commit in to address this one! @syedahsn |
Document the behaviour and interaction between exec_config and run_task_kwargs config
2a7cb6b
to
93f7b6f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like all concerns have been addressed 👍
commit 58279b8 Author: Satish Ch <satishchinthanippu@gmail.com> Date: Sat Feb 10 00:12:42 2024 +0530 Update teradata.py commit ed75e3e Author: Satish Ch <satishchinthanippu@gmail.com> Date: Sat Feb 10 00:09:24 2024 +0530 D401 support in Teradata Provider commit f56bede Merge: e859a1d 00ed467 Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 23:58:15 2024 +0530 Merge remote-tracking branch 'upstream/main' into pr_teradata_release_1.0.0 commit 00ed467 Author: Vincent <97131062+vincbeck@users.noreply.github.com> Date: Fri Feb 9 13:13:36 2024 -0500 D401 support in fab provider (apache#37283) commit 48bfb1a Author: Niko Oliveira <onikolas@amazon.com> Date: Fri Feb 9 08:43:32 2024 -0800 Merge all ECS executor configs following recursive python dict update (apache#37137) Also document the behaviour and interaction between exec_config and run_task_kwargs config commit 8317ed9 Author: Amogh Desai <amoghrajesh1999@gmail.com> Date: Fri Feb 9 20:17:38 2024 +0530 Updating the README and visuals for breeze build-docs (apache#37276) commit 17945fc Author: Kalyan <kalyan.ben10@live.com> Date: Fri Feb 9 20:16:33 2024 +0530 D401 fixes in Pinecone provider (apache#37270) commit ab9e2e1 Author: Kalyan <kalyan.ben10@live.com> Date: Fri Feb 9 20:15:31 2024 +0530 fix: D401 lint issues in airflow core (apache#37274) commit 7835fd2 Author: Jarek Potiuk <jarek@potiuk.com> Date: Fri Feb 9 14:59:33 2024 +0100 The fix-ownership command missed --rm flag and left dangling containers (apache#37277) Fixes: apache#37269 commit e859a1d Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 19:21:28 2024 +0530 Update teradata.py commit d1c08e1 Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 19:18:37 2024 +0530 Update teradata.py commit e5ac0ec Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 05:41:42 2024 -0800 static check issue is fixed commit ce490f7 Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 19:10:36 2024 +0530 static format issue fixed commit f9498c5 Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 05:36:36 2024 -0800 static check issue is fixed commit 77bddae Author: Satish Ch <satishchinthanippu@gmail.com> Date: Fri Feb 9 18:23:18 2024 +0530 common sql unit test failure fixed commit 9f4f208 Author: Aleksey Kirilishin <54231417+avkirilishin@users.noreply.github.com> Date: Fri Feb 9 15:53:04 2024 +0300 Fix the bug that affected the DAG end date. (apache#36144) commit 0f8dfeb Author: Sudipto Baral <sudiptobaral.me@gmail.com> Date: Fri Feb 9 06:54:23 2024 -0500 fix: update hyperlink to the new documentation section for local virtualenv setup. (apache#37272) Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>
This PR changes how
exec_config
works for the ECS executor. Previously theexec_config
was only applied to item zero of thecontainerOverrides
list (the container that receives the Airflow task command). This made it impossible for folks to override any configuration at the base oroverrides
level of the runTask kwargs (see docs for possible configuration here).The new strategy is to treat the
exec_config
as an entire run task kwargs config and do a recursive dict update. This allows folks to set any value from a task or DAG.This behaviour change is allowed due to the feature still being experimental.
fixes #35490
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.