[BUG] Unable to use saltcheck with a map.jinja

[pgporada]

Description

[root@saltmaster01 ~]# salt 'minion' saltcheck.run_state_tests whatever.whatever
whatever:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
        output = template.render(**decoded_context)
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "<template>", line 1, in top-level template code
      File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 193, in get_source
        raise TemplateNotFound(template)
    jinja2.exceptions.TemplateNotFound: whatever/map.jinja
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1676, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 431, in run_state_tests
        stl.load_test_suite()
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 972, in load_test_suite
        self._load_file_salt_rendered(myfile)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 980, in _load_file_salt_rendered
        tests = _render_file(filepath)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 507, in _render_file
        rendered = __salt__['slsutil.renderer'](file_path)
      File "/usr/lib/python3.6/site-packages/salt/modules/slsutil.py", line 182, in renderer
        **kwargs
      File "/usr/lib/python3.6/site-packages/salt/template.py", line 101, in compile_template
        ret = render(input_data, saltenv, sls, **render_kwargs)
      File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 70, in render
        **kws)
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 169, in render_tmpl
        output = render_str(tmplstr, context, tmplpath)
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 443, in render_jinja_tmpl
        trace=tracestr)
    salt.exceptions.SaltRenderError: Jinja error: whatever/map.jinja
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
        output = template.render(**decoded_context)
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "<template>", line 1, in top-level template code
      File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 193, in get_source
        raise TemplateNotFound(template)
    jinja2.exceptions.TemplateNotFound: whatever/map.jinja
    
    ; line 1
    
    ---
    {% from "whatever/map.jinja" import whatever with context %}    <======================
    
    [...]
    ---
ERROR: Minions returned with non-zero exit code

Setup

salt/
  base/
    whatever/
      saltcheck/
        whatever.tst
    whatever.sls
    map.jinja
    defaults.yaml

The map.jinja contains

{% import_yaml 'whatever/defaults.yaml' as default_settings %}

{% set init_map = salt['grains.filter_by']({
        'CentOS Linux-7': {
          'process': {
            'servicemode': '644',
          }
        }
  }
  , grain="osfinger"
  , merge=salt['pillar.get']('whatever:lookup'))
%}

{% do default_settings.whatever.update(init_map) %}

{% set whatever = salt['pillar.get'](
        'whatever',
        default=default_settings.whatever,
        merge=True
    )
%}

The whatever.tst file contains

{% from "whatever/map.jinja" import whatever with context %}

verify_package_version:
  module_and_function:  pkg.version
  args:
    - {{ whatever.pkg }}
  kwargs:
  assertion: assertEqual
  expected_return: {{ whatever.pkg }}-{{ whatever.version }}

Expected behavior
I should be able to use the map.jinja from a formula in a saltcheck test.

Versions Report

Salt Version:
           Salt: 3000.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.7.1908 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.18.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

Additional context
Add any other context about the problem here.

[garethgreenaway]

@mchugh19 Any insights you could offer on this one?

[mchugh19]

@garethgreenaway map files should be supported. There was a conversation about how best to handle this in #53024. It really comes down to the saltcheck running files through the renderer, and figuring out what's needed, then caching that. Map files were very much tested, so I'd expect this to work, but I don't see map files included in the integration test.

[mchugh19]

I just tried to replicate by cloning the ntp formula and copied the ntp directory to /srv/salt. I then added ntp/saltcheck-tests/init.tst with the content:

{% from "ntp/map.jinja" import ntp with context %}

check_installed:
  module_and_function: pkg.version
  args:
    - {{ ntp.client }}
  assertion: assertNotEmpty

{# check that our custom ntp server exists in the ntp.conf file #}
ensure_correct_file:
  module_and_function: file.contains
  args:
    - {{ ntp.ntp_conf }}
    - 'pool 0.debian.pool.ntp.org'
  assertion: assertTrue

check_ntp_running:
  module_and_function: service.status
  args:
    - {{ ntp.service }}
  assertion: assertTrue

As you can see, this test uses the formula's map.jinja file.

I am then able to run saltcheck with this test:

# salt-call --local saltcheck.run_state_tests ntp
local:
    |_
      ----------
      ntp:
          ----------
          check_installed:
              ----------
              duration:
                  1.2897
              status:
                  Pass
          ensure_correct_file:
              ----------
              duration:
                  1.2573
              status:
                  Pass
          check_ntp_running:
              ----------
              duration:
                  1.2793
              status:
                  Pass
    |_
      ----------
      TEST RESULTS:
          ----------
          Execution Time:
              3.8263
          Failed:
              0
          Missing Tests:
              0
          Passed:
              3
          Skipped:
              0

It also works with master/minion

# rm -r /var/cache/salt/minion/files/base/ntp/
# salt 'saltdev' saltcheck.run_state_tests ntp
saltdev:
   ...
      TEST RESULTS:
          ----------
          Execution Time:
              3.7914
          Failed:
              0
          Missing Tests:
              0
          Passed:
              3
          Skipped:
              0

Thus, my testing seems to show that map.jinja files are rendered just fine, and I'm not able to replicate. The reported error makes it seem that the file doesn't actually exist

      File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 193, in get_source
        raise TemplateNotFound(template)
    jinja2.exceptions.TemplateNotFound: whatever/map.jinja

@pgporada Can you double check the filename and paths? Or is there any special about your testcase that I didn't replicate like gitfs or salt-ssh?

[mchugh19]

I've also tried to replicate with the given whatever files:

/srv/salt/whatever/
├── defaults.yaml
├── map.jinja
├── saltcheck-tests
│   └── whatever.tst
└── whatever.sls

whatever/map.jinja - Note updated for my test host of Debian-10

{% import_yaml 'whatever/defaults.yaml' as default_settings %}

{% set init_map = salt['grains.filter_by']({
        'Debian-10': {
          'process': {
            'servicemode': '644',
          }
        }
  }
  , grain="osfinger"
  , merge=salt['pillar.get']('whatever:lookup'))
%}


{% do default_settings.whatever.update(init_map) %}

{% set whatever = salt['pillar.get'](
        'whatever',
        default=default_settings.whatever,
        merge=True
    )
%}

whatever/whatever.sls

{% from "whatever/map.jinja" import whatever with context %}

always-passes-with-any-kwarg:
  test.nop:
    - name: whatever
    - something: else

whatever/saltcheck-tests/whatever.tst

{% from "whatever/map.jinja" import whatever with context %}

verify_package_version:
  module_and_function:  pkg.version
  args:
    - {{ whatever.pkg }}
  kwargs:
  assertion: assertEqual
  expected_return: {{ whatever.pkg }}-{{ whatever.version }}

whatever/defaults.yml

whatever:
  pkg: "something"
  version: "1.2.3"

Given this file structure, I am able to display the map.jinja contents:

# salt-call --local jinja.load_map whatever/map.jinja whatever
local:
    ----------
    pkg:
        something
    process:
        ----------
        servicemode:
            644
    version:
        1.2.3

So I can validate that the map.jinja is being successfully rendered.

Then I'm able to run saltcheck for the given state

# salt-call --local saltcheck.run_state_tests whatever.whatever
local:
    |_
      ----------
      whatever.whatever:
          ----------
          verify_package_version:
              ----------
              duration:
                  1.2199
              status:
                  Fail: something-1.2.3 is not equal to 
    |_
      ----------
      TEST RESULTS:
          ----------
          Execution Time:
              1.2199
          Failed:
              1
          Missing Tests:
              0
          Passed:
              0
          Skipped:
              0

[cingeyedog]

It appears that importing map.jinja from saltcheck-tests, doesn't honor saltenv.

# cat /etc/salt/master.d/master.conf
log_level_logfile: debug

file_roots:
  base:
    - /srv/salt
  dev:
    - /srv/salt-dev

Here is the directory structure.

# tree salt-dev/
salt-dev/
└── grub
    ├── init.sls
    ├── map.jinja
    └── saltcheck-tests
        └── init.tst

I am able to run salt -C "test-minion" state.apply grub saltenv=dev without issues.

However, when I run salt -C "test-minion" saltcheck.run_state_tests grub saltenv=dev, I get the same errors as described above.

# salt -C "test-minion" saltcheck.run_state_tests grub saltenv=dev
test-minion:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
        output = template.render(**decoded_context)
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "<template>", line 1, in top-level template code
      File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 193, in get_source
        raise TemplateNotFound(template)
    jinja2.exceptions.TemplateNotFound: grub/map.jinja

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1676, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 431, in run_state_tests
        stl.load_test_suite()
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 972, in load_test_suite
        self._load_file_salt_rendered(myfile)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 980, in _load_file_salt_rendered
        tests = _render_file(filepath)
      File "/usr/lib/python3.6/site-packages/salt/modules/saltcheck.py", line 507, in _render_file
        rendered = __salt__['slsutil.renderer'](file_path)
      File "/usr/lib/python3.6/site-packages/salt/modules/slsutil.py", line 182, in renderer
        **kwargs
      File "/usr/lib/python3.6/site-packages/salt/template.py", line 101, in compile_template
        ret = render(input_data, saltenv, sls, **render_kwargs)
      File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 70, in render
        **kws)
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 169, in render_tmpl
        output = render_str(tmplstr, context, tmplpath)
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 443, in render_jinja_tmpl
        trace=tracestr)
    salt.exceptions.SaltRenderError: Jinja error: grub/map.jinja
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
        output = template.render(**decoded_context)
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "<template>", line 1, in top-level template code
      File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 193, in get_source
        raise TemplateNotFound(template)
    jinja2.exceptions.TemplateNotFound: grub/map.jinja

    ; line 1

    ---
    {%- from "grub/map.jinja" import grub_path with context %}    <======================

    Verify the /etc/default/grub is missing "rhgb quiet":
      module_and_function: file.contains
      args:
        - /etc/default/grub
    [...]
    ---

However, if i change the the file_root for dev to point to /srv/salt/dev and move my test state directory to /srv/salt/dev/grub:

# cat /etc/salt/master.d/master.conf
log_level_logfile: debug

file_roots:
 base:
   - /srv/salt
 dev:
   - /srv/salt/dev

AND I change the saltcheck-test/init.tst jinja import from the base:

# head -1 dev/grub/saltcheck-tests/init.tst
{%- from "dev/grub/map.jinja" import grub_path with context %}

Everything works as expected

# salt -C "test-minion" saltcheck.run_state_tests grub saltenv=dev
test-minion:
    |_
      ----------
      grub:
          ----------
          Verify the /boot/grub2/grub.cfg is missing "rhgb quiet":
              ----------
              duration:
                  0.484
              status:
                  Pass
          Verify the /etc/default/grub is missing "rhgb quiet":
              ----------
              duration:
                  5.6955
              status:
                  Pass
    |_
      ----------
      TEST RESULTS:
          ----------
          Execution Time:
              6.1795
          Failed:
              0
          Missing Tests:
              0
          Passed:
              2
          Skipped:
              0

Here is my versions-report

# salt --versions-report
Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: 1.14.0
       cherrypy: unknown
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: 0.28.4
       M2Crypto: 0.33.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: 3.9.7
         pygit2: 1.0.3
         Python: 3.6.8 (default, Nov 27 2019, 14:21:59)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: oracle 7.8
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.12.1.el7.x86_64
         system: Linux
        version: Oracle Linux Server 7.8

[mchugh19]

Thanks for the info @cingeyedog

I just pushed a fix for this in #57557. Sodium is already in a release freeze, so this is unlikely to make it in time, but you get the fixed version from that PR and place it into your salt states/_modules directory (and run saltutl.sync_all) to correct your existing setup.

It's a one line fix, and the saltcheck file itself should be able to be backported to any older release of salt.

[cingeyedog]

Thank you @mchugh19

Verified the fix provided resolved the issue.

[pgporada]

I can confirm that this fixed my problem too.