[AIRFLOW-3362] [WIP]Template to support jinja2 native python types

[ryanyuan]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow-3362 issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal(AIP).

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[ashb]

We should just update the minimum version of Jinja we require instead.

[ryanyuan]

We should just update the minimum version of Jinja we require instead.

Updating the minimum version of Jinja cannot resolve the #4770 (list as templated field issue). Referring to your suggestion in that PR, we can either import jinja2.NativeEnvironment or convert specific templated fields into list type in those operators.

[ashb]

Sorry, I mean update jinja version to one that always has NativeEnvironment - so we don't have to have the except ImportError.

[ryanyuan]

You are right! Thanks for the suggestion!

[mik-laj]

Have you analyzed the use of literal_eval? It is currently used in some operators

airflow/airflow/contrib/operators/jenkins_job_trigger_operator.py

Lines 138 to 139 in 2c255e9

	import ast
	self.parameters = ast.literal_eval(self.parameters)

airflow/airflow/operators/docker_operator.py

Lines 244 to 248 in 067a1e3

	if self.command is not None and self.command.strip().find('[') == 0:
	commands = ast.literal_eval(self.command)
	else:
	commands = self.command
	return commands

[ryanyuan]

@mik-laj Yes, in #4770, I used ast.literal_eval() to do the job, but on a local level. And I was suggested to do it, in #4848, on a global level using Jinja2's NativeEnvironment.

Now I am having a bottleneck, which is Jinja2 trying to get the templated field's value of date type.
E.g.
Templated value is 2019-11-11. And Jinja2 sees it as subtraction and returns 1997.

So I am thinking to do ast.literal_eval() inside airflow.models' render_template() function. @ashb

[ashb]

K, so full Native Environment won't work, that's a shame.

One change I'd suggest to #4770 is to at least override the render_template function on the operator, rather than having to do it in the execute function.

[ashb]

Closing this PR as the Native type support does funny things on dates:

Templated value is 2019-11-11. And Jinja2 sees it as subtraction and returns 1997.

Shame