-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement EagerFromTag and EagerImportTag #560
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Any questions or concerns before I merge this in? Again, this logic is only impacting Eager Execution, so any changes, including retroactive are easy to make. |
Joeoh
reviewed
Dec 17, 2020
Joeoh
approved these changes
Dec 17, 2020
boulter
approved these changes
Dec 18, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #532
This PR finishes the logic for importing with either a from tag or an import tag. Over half the additions here are tests, but the logic is somewhat complex/lengthy so I'll do my best to explain it well.
EagerImportTag
The import tag is mainly used to import macros from another file, but it also supports importing values set from another file. The deferred importing of both macros and values need to be handled. The goal is to make the output from eager execution not depend on any other files. So the data from the imported file needs to be serialised in some way to the output from eager execution if those values are dependent on deferred values.
Import tags can either use an alias or not. A tag that imports with an alias, like
{% import x as y %}
stores all the macros and values fromx
likey.z
. Without an alias like{% import x %}
would store a macro declared inx
,foo()
, as simplyfoo()
(as opposed toy.foo()
). This is how imports work normally so the output when an import gets deferred needs to be able to populate these alias maps that can be multi-leveled (as there can be multiple levels of importing). This is done by tracking the full import resource alias name so that given 3 files "file1.jinja", "file2.jinja", "file3.jinja" as:Would result in a context map that looks like:
{'x': {'y': {'bar': 1234}}}
with the value of bar accessable with:{{ x.y.bar }}
If instead,
bar
depended on a deferred value, the simplified output from executing "file3.jinja" would look something like:This is achieved with a process that I'll briefly describe. When an import tag is executed, I now add an additional meta value on the context:
Context.IMPORT_RESOURCE_ALIAS_KEY
is the key for the value which marks the full import alias. In the above example, while executing 'file1.jinja', this would bex.y
. When executing a set tag, in addition to executing/reconstructing the set tag, a do-update tag is constructed if something depends on a deferred value. When executed during a second pass, this will update the alias map to hold the desired value, while allowing any other tokens that may be deferred in the imported file to be able to execute properly the second time.After the import file is interpreted, that child context gets integrated with the (parent) context in which the file was imported. If the import was done using an alias, the alias map is copied to the parent context. If it's not done using an alias, then the child bindings are simply copied over. If anything was deferred during the import, then a do-update tag is created that defines the alias map (which is essentially a copy of the child context's bindings) that can be loaded during a second pass.
Also, there are macros that get imported. Without an alias, they will get put as global macros, but with an alias, they will get loaded onto the alias map as what I refer to as "localMacros". They're called with
alias.macro()
, like in the normal import tag functionality. See #547 for the deferred macro logic when a macro function also needs to be deferred.The caveat with this approach is that name clashes from deferred values don't get handled properly in one case:
This is an extremely niche case and I couldn't think of a way to solve it. I highly doubt there are any real-world usages that this would impact.
EagerFromTag
From tags are much easier because you explicitly specify which variables or macros to import from the file. The file which is being imported gets executed without anything really special. From tags also allow aliases for individual variables or macros, but these are different as they are simply renaming the variable or macro. In the case where a value needs to be deferred, then a set tag is created that just sets a variable with "new_name" to the value of "old_name" like:
{% set new_name = old_name %}
.There is a similar niche case that exists for this logic as well:
x
) is set on the context before a from tag{% from file import x as y %}
)file
, the value ofx
can not be resolved, and it must be a deferred valuex
, but it is now deferred and will have the same value asy
when run through a second pass.cc @jboulter @Joeoh this should be the remaining logic that's needed for eager execution to be usable