[BUG] salt-ssh vs salt installation problems 2016.3.8

[apex-omontgomery]

Your setup

Formula commit hash / release tag

Targetting master branch

# master config
fileserver_backend:
  - roots
  - gitfs

gitfs_provider: pygit2
gitfs_base: master

gitfs_remotes:
  - https://github.com/saltstack-formulas/docker-formula.git

Versions reports (master & minion)

Currently running 2019.2.0 on salt-master and 2016.11.10 to test out compatability.
This also occcurs on 2016.3.8

Master versions

root@salt-master:/alldata/salt# salt --versions-report
Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: 1.12.3
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.27.4
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.27.3
         Python: 3.5.2 (default, Oct  8 2019, 13:06:37)
   python-gnupg: Not Installed
         PyYAML: 5.1.2
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.15.0-60-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Master versions

root@salt-minion-1:/# salt-minion --versions-report
Salt Version:
           Salt: 2016.11.10

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Oct  8 2019, 14:14:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.15.0-60-generic
         system: Linux
        version: Ubuntu 16.04 xenial

### Pillar / config used

# pillar/base/init.sls
docker:
  install_pypi_pip: True
    pkg:
    version: 1.13.1
    use_upstream_app: False

---

Bug details

Describe the bug

salt-ssh \* -i state.apply base


Summary for salt-minion-0
-------------
Succeeded: 12 (changed=8)
Failed:     0
-------------

Properly installs docker service. 

When I run it again, I get a failure. 

Summary for salt-minion-0

Succeeded: 12 (changed=8)
Failed: 0

   - Rendering SLS 'base:docker' failed: Jinja error: docker/map.jinja
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 989, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 754, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
          raise value.with_traceback(tb)
        File "<template>", line 5, in top-level template code
        File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 140, in get_source
          raise TemplateNotFound(template)
      jinja2.exceptions.TemplateNotFound: docker/map.jinja

      ; line 5

      ---
      # -*- coding: utf-8 -*-
      # vim: ft=sls
      {#- Get the `tplroot` from `tpldir` #}
      {%- set tplroot = tpldir.split('/')[0] %}
      {%- from tplroot ~ "/map.jinja" import docker with context %}    <======================
      {%- set sls_package_install = tplroot ~ '.install' %}
      {%- set sls_macapp_install = tplroot ~ '.macosapp' %}

      include:
        - {{ sls_package_install if not docker.pkg.use_upstream_app else sls_macapp_install }}
      [...]
      ---

I also see this error, sometimes.

    Comment: Service salt-minion is already enabled, and is dead

But when I do:

salt '*' state.apply base

Summary for salt-minion-2
-------------
Succeeded: 11 (changed=1)
Failed:     0
-------------
Total states run:     11
Total run time:   29.083 s

Steps to reproduce the bug

1. salt-master : 2019.2.0 salt-master: 2016.3.8
2. install docker using salt-ssh or salt
3. Run base state with salt '*' state.apply base -> no error
4. Run base state with salt-ssh '*' -i state.apply base -> error

Expected behaviour

formula continues to work.

Attempts to fix the bug

Additional context

This seems related to the

service.running:
  enable: True

As that was the error that was occuring before I upgrade the salt-minion to 2016.11.10.
That was fixed by removing that keyvalue pair for the state in my own internal formulas.

I'm not really sure that this is a docker-formula issue, but in general to the above. I'd like to know how to deal with backwards compatability and which salt-master/ salt-minions this is expected to work with.

[myii]

@wimo7083 Thanks for the report. A couple of initial comments.

There are general (i.e. not specific to SaltStack Formulas), known issues with salt-ssh. One example that we've encountered a number of times is saltstack/salt#51605.

Looking back at the history of map.jinja, it looks like it has had it's share of salt-ssh issues as well: https://github.com/saltstack-formulas/docker-formula/commits/master/docker/map.jinja.

Perhaps some of those issues can be ironed out by moving to the format laid out in the template-formula, which has paid full consideration to salt-ssh along the way. You might find that solves things if you're able to modify that file accordingly. Otherwise, please ping here again and I'll try to provide you an appropriate update.

A separate note is regarding 2016.3 -- this is no longer a supported version of Salt. There are three major versions supported at any one time, meaning that 2017.7 is the oldest supported version right now. Formulas are designed and tested with that in mind, so may not work for anything older than that.

[noelmcloughlin]

Fixed in merged #256