Can't use tpldir or slspath in template file

[hatifnatt]

Can't use tpldir or slspath in template file

I want to load defaults with construction like this

{%- import_yaml slspath ~ "/defaults.yaml" as defaults %}

Which work flawless in sls state file. But fails in template file. On #salt IRC channel I received a recommendation to use 'tpldir' variable instead, but it doesn't work as well.

Setup

init.sls

{%- import_yaml slspath ~ "/defaults.yaml" as defaults %}
config:
  file.managed:
    - name: "/tmp/test.cfg"
    - source: salt://{{ slspath }}/test.cfg.jinja
    - template: jinja

test.cfg.jinja

{%- import_yaml tpldir ~ "/defaults.yaml" as defaults %}
opt1 = var1
opt2 = var2

With setup like this I receive error:

          ID: config
    Function: file.managed
        Name: /tmp/test.cfg
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1746, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1704, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 1821, in managed
                  **kwargs
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 4340, in check_managed_changes
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 3834, in get_managed
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 178, in render_tmpl
                  output = render_str(tmplstr, context, tmplpath)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 386, in render_jinja_tmpl
                  buf=tmplstr)
              SaltRenderError: Jinja variable 'tpldir' is undefined
     Started: 18:20:39.233032
    Duration: 52.652 ms
     Changes:

Expected result is 'tpldir' will contain path to current template file.

Versions Report

Salt Version:
           Salt: 2016.11.4

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.9 (default, Jun 29 2016, 13:08:31)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.8
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Linux
        version: debian 8.8

[gtmanfred]

When they said to use tpldir, did they mention where in salt that was documented? because i cannot find it in the documents at all :/ It might be just a jinja thing.

Can you try using

{%- import_yaml "defaults.yaml" as defaults %} ? i wonder if it will allow relative paths.

I am looking into this further.

Thanks,
Daniel

[gtmanfred]

ahh, i see it referenced in salt.utils.jinja

[hatifnatt]

I have tried {%- import_yaml "defaults.yaml" as defaults %} my bad I doesn't mentioned it straight away, but it doesn't work too.

Comment: An exception occurred in this state: Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1746, in call
            **cdata['kwargs'])
          File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1703, in wrapper
            return f(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 1793, in managed
            **kwargs
          File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 4299, in check_managed_changes
            **kwargs)
          File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 3793, in get_managed
            **kwargs)
          File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 178, in render_tmpl
            output = render_str(tmplstr, context, tmplpath)
          File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 415, in render_jinja_tmpl
            trace=tracestr)
        SaltRenderError: Jinja error: defaults.yaml
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl
            output = template.render(**decoded_context)
          File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
            return self.environment.handle_exception(exc_info, True)
          File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
            reraise(exc_type, exc_value, tb)
          File "<template>", line 3, in top-level template code
          File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 135, in get_source
            raise TemplateNotFound(template)
        TemplateNotFound: defaults.yaml

[hatifnatt]

And about documentation, it isn't documented anywhere, only in code and in other issues. Mostly discussed here #4348.

[gtmanfred]

ok, let me take a look closer, because I see where it should be injecting it into the template environment in salt.utils.jinja.

[gtmanfred]

Thanks for reporting this. I have replicated the issue, and looking at the source this should definitely work.

We attempt to add the tpldir stuff into the environment for jinja files here https://github.com/saltstack/salt/blob/2016.11/salt/utils/jinja.py#L106-L113

But it doesn't look like the jinja templates actually get run through there, instead they go through https://github.com/saltstack/salt/blob/develop/salt/utils/templates.py#L109-L129

And because the template is not an sls file, we never add the tpldir information to the template.

So, even though the original commit that adds this wanted to add what you are requesting, it doesn't work. And from testing all the way back to that commit, it looks like it never worked.

I am tagging this as a bug for our core team. Thanks for reporting. If you wanted to take a shot at fixing this, we would greatly appreciate it. Here are the jinja docs which should be how our cache loader is loaded and used for loading templates. http://jinja.pocoo.org/docs/2.9/api/#jinja2.BaseLoader.get_source

Thanks,
Daniel

[hatifnatt]

@gtmanfred thanks for analysis, I had the same assumption. But this assumption is best of my Python skills :) I will patiently wait for fix from Salt team.

[lorengordon]

@gtmanfred I wonder if I was having the same problem when trying to use tpldir in a winrepo pkg definition?

#29063

[gtmanfred]

That would not surprise me at all.

…

On Fri, May 12, 2017 at 5:29 AM, Loren Gordon ***@***.***> wrote: @gtmanfred <https://github.com/gtmanfred> I wonder if I was having the same problem when trying to use tpldir in a winrepo pkg definition? #29063 <#29063> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41195 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoWUVAgvXGY8lDNW9FNXG0Ra4AeB_ks5r5EKRgaJpZM4NYSEh> .

[mkotsbak]

Any progress here?

[mkotsbak]

Workaround described here: saltstack-formulas/mysql-formula#184 (comment)

[myii]

Where map.jinja is imported with context, here's another workaround:

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{#- Start imports as #}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}
{%- import_yaml tplroot ~ "/osfingermap.yaml" as osfingermap %}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[myii]

This is still present in 2019.2.2. Please keep this issue open.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[max-arnold]

@myii (salt-formulas org maintainer) informed me about a problem with the tpldir, slspath, tplfile, and tpldot template variables when using the new map.jinja troubleshooting module introduced in #55253.

I'm not 100% familiar with the code (I just backported it to Salt Neon), but on a first glance it looks like the problem is caused by using the salt.template.compile_template_str in modules/jinja.py. To evaluate a map.jinja file it renders the following inline template string that imports the specified map.jinja:

{{% from "{path}" import {value} with context %}}
{{{{ {value} | tojson }}}}

(1) The above snippet gets saved into a tempfile (something like /tmp/__salt.tmp.f1t0s2c_). That is probably the reason why these path variables aren't defined.

(2) The second part of the problem is that a map.jinja could be imported in different sls files that are located in a formula root or in a subfolder. This produces different values of the above mentioned template variables.

(3) And the final issue is that these variables have different values depending on the presence of with context import clause.

Below is a reproducible test case:

% tree formula

formula
├── init.sls
├── map.jinja
└── subfolder
    └── init.sls

init.sls (both files are the same):

{% do salt.log.warning('INSIDE INIT:') %}
{% do salt.log.warning('VAR tpldir="{}"'.format(tpldir)) %}
{% do salt.log.warning('VAR slspath="{}"'.format(slspath)) %}
{% do salt.log.warning('VAR tplfile="{}"'.format(tplfile)) %}
{% do salt.log.warning('VAR tpldot="{}"'.format(tpldot)) %}

{% do salt.log.warning('WITHOUT CONTEXT:') %}
{% from "formula/map.jinja" import defaults %}

{% do salt.log.warning('WITH CONTEXT:') %}
{% from "formula/map.jinja" import defaults with context %}

map.jinja:

{% do salt.log.warning('INSIDE MAP:') %}
{% do salt.log.warning('VAR tpldir="{}"'.format(tpldir)) %}
{% do salt.log.warning('VAR slspath="{}"'.format(slspath)) %}
{% do salt.log.warning('VAR tplfile="{}"'.format(tplfile)) %}
{% do salt.log.warning('VAR tpldot="{}"'.format(tpldot)) %}

{% set defaults = {} %}

Below is how issues (2) and (3) mainfest themselves:

% sudo salt-call state.apply formula -l warning

[WARNING ] INSIDE INIT:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/init.sls"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITHOUT CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITH CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula"
[WARNING ] VAR tplfile="formula/init.sls"
[WARNING ] VAR tpldot="formula"

% sudo salt-call state.apply formula.subfolder -l warning

[WARNING ] INSIDE INIT:
[WARNING ] VAR tpldir="formula/subfolder"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/subfolder/init.sls"
[WARNING ] VAR tpldot="formula.subfolder"
[WARNING ] WITHOUT CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"
[WARNING ] WITH CONTEXT:
[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula/subfolder"
[WARNING ] VAR slspath="formula/subfolder"
[WARNING ] VAR tplfile="formula/subfolder/init.sls"
[WARNING ] VAR tpldot="formula.subfolder"

And below is the issue (1) with the new jinja.py module:

% sudo salt-call jinja.load_map formula/map.jinja defaults

[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="."
[WARNING ] VAR slspath=""
[WARNING ] VAR tplfile=""
[WARNING ] VAR tpldot=""

I made an experimental patch to modules/jinja.py that improves the behavior:

diff --git a/salt/modules/jinja.py b/salt/modules/jinja.py
index eb23991c37..f5760b5767 100644
--- a/salt/modules/jinja.py
+++ b/salt/modules/jinja.py
@@ -52,12 +52,16 @@ def load_map(path, value):
         {{% from "{path}" import {value} with context %}}
         {{{{ {value} | tojson }}}}
         '''.format(path=path, value=value))
-    return salt.template.compile_template_str(
-        tmplstr,
+    return salt.template.compile_template(
+        ':string:',
         salt.loader.render(__opts__, __salt__),
         __opts__['renderer'],
         __opts__['renderer_blacklist'],
-        __opts__['renderer_whitelist'])
+        __opts__['renderer_whitelist'],
+        input_data=tmplstr,
+        tmplpath=path,
+        sls=path
+    )

% sudo salt-call jinja.load_map formula/map.jinja defaults

[WARNING ] INSIDE MAP:
[WARNING ] VAR tpldir="formula"
[WARNING ] VAR slspath="formula/map/jinja"
[WARNING ] VAR tplfile="formula/map.jinja"
[WARNING ] VAR tpldot="formula"

However, the slspath is obviously incorrect.

One potential workaround is to introduce the tplroot variable (see #51814, however if you place a formula into a deeply nested dir (e.g., blah/blah/blah/blah/template-formula), the tplroot variable will be just blah and all the import paths inside the formula will be broken.

TLDR: salt-formulas need path-independent ways to reference map.jinja files (and also *.yaml and *.json data files) inside of *.sls files. Also we need a simple way to debug these files in isolation. Plus, all template variables should be documented: #50925

CC: @terminalmage You wrote the original jinja.py troubleshooting module and have much more experience in Salt renderer subsystem. Maybe you can provide some clues on how to fix this? Also, are there any reasons why you used the salt.template.compile_template_str function instead of salt.template.compile_template?

[terminalmage]

@max-arnold compile_template_str is designed to take a string, while compile_template is designed to take a path to a template file.

With regard to tpldir, tpldot, and tplfile, these are also SLS-specific. That is, they are different depending on what SLS file is being processed. When compile_template is operating on a template, it is not aware of the URL used to fetch the template, just the contents of the template. So, outside of the context of an SLS file (which has a known relative path), there is no good way for the templating system to know what these values should be. And also, keep in mind that these values are different depending on the path of the SLS file!

When I need to use these variables in a jinja template, I just pass them in the context argument, like so:

/etc/foo.conf:
  file.managed:
    - source: salt://apps/foo/foo.conf.jinja
    - template: jinja
    - context:
        tpldir: {{ tpldir }}

[myii]

@terminalmage The discussion with @max-arnold started up again when I realised that the PR that was merged into develop needed to ported to master: #51814.

Just quoting part of the description here:

Working on making SaltStack Formulas portable, find the need for a
common point of reference. Comparing the import of map.jinja with and
without context:

tpldata	without context	with context
tplfile	template/map.jinja	template/pkg/install.sls
tpldir	template	template/pkg
tpldot	template	template.pkg
tplroot	template	template

With tplroot, it is possible to use a single point of reference for
all paths, ...

We've been using this across the formulas now for some time using:

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}

It's works with .sls files in the top-directory or nested at any level in the sub-directories, which hasn't been the case with tpldir. It's also allowed formulas to be portable, so that the formula directory can be renamed as desired by the end user.

One problem was identified by @max-arnold above, though:

One potential workaround is to introduce the tplroot variable (see #51814, however if you place a formula into a deeply nested dir (e.g., blah/blah/blah/blah/template-formula), the tplroot variable will be just blah and all the import paths inside the formula will be broken.

I suppose one way around this it to allow formula users to provide an override for tplroot via. the config/pillar, should they need it. In my opinion, having tplroot directly available is still a useful default, where users are following the instructions in the official documentation.

What's your opinion about how this should proceed? Should tplroot be introduced to the master branch or should we go back to using tpldir? If the latter, what's the best way to use this when there are .sls files at multiple levels within a formula's directory structure?

[sagetherage]

@saltstack/docs-working-group

[terminalmage]

@myii Sorry, I missed your question from several months ago. It seems that your tplroot PR was ported to master already, which I think was the right move.

[barbaricyawps]

For those of you who are watching this thread, can anyone tell me if the PR linked in this issue resolved this documentation issue or not? If so, I'll close it out.

[baby-gnu]

Hello @barbaricyawps

For those of you who are watching this thread, can anyone tell me if the PR linked in this issue resolved this documentation issue or not? If so, I'll close it out.

Sorry, which PR are you speaking about? So many things are linked to this.

Regards.