Amazon EKS example DAG raises warnning

[eladkal]

Body

Importing package: airflow.providers.amazon.aws.example_dags
[2022-05-17 15:53:10,675] {eks.py:287} WARNING - The nodegroup_subnets should be List or string representing Python list and is {{ dag_run.conf['nodegroup_subnets'] }}. Defaulting to []

airflow/airflow/providers/amazon/aws/example_dags/example_eks_templated.py

Line 82 in dec05fb

nodegroup_subnets="{{ dag_run.conf['nodegroup_subnets'] }}",

example for waning raising in the CI:
https://github.com/apache/airflow/runs/6474027737?check_suite_focus=true#step:12:1010

We should fix it. The example DAG should not yield warnings.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Yeah. we should remove those warnings!

[Dark-Knight11]

Hi, I would like to work on this issue. Can you please assign it to me?

[potiuk]

Done

[Dark-Knight11]

Hi, I looked into the issue and was confused about what to do.
ast.literal_eval is not able to process this string {{ dag_run.conf['nodegroup_subnets'] }}
I also looked in other files where literal_eval is used. DockerOperator uses it but I'm getting errors there as well.
Can you please help me with correct string format or do we need to make changes in EksCreateNodegroupOperator

[josh-fell]

The issue is that nodegroup_subnets is a template field for the operator and the string evaluation on that arg is happening as part of the constructor. We've been slowly finding more and more of these instances and pushing them out of __init__(), but template fields are not rendered (i.e. retain their stringified Jinja value) until just before the execute() method is called. So string evaluations on template fields should not happen until the execute() scope. This would also generally emit a warning if nodegroup_subnets was an XComArg as well.

Moving that evaluation section to the scope of the execute() method should do the trick and be functionally correct.