[BUG] Salt minion masterless requires pkg.refresh_db to be executed twice

[amalaguti]

Description
WIdows minion in masterless setup with a new software definition file in local default win_repo_ng dir path, requires to execute pkg.refresh_db twice to get it updated.

C:\Users\adrian>salt-call --local state.sls tanium.win_repo-nessusagent
local:
----------
          ID: Download_PkgMgr_StateFile
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\srv\salt\win\repo-ng\nessus-agent.sls
      Result: True
     Comment: File C:\ProgramData\Salt Project\Salt\srv\salt\win\repo-ng\nessus-agent.sls updated
     Started: 17:42:09.843687
    Duration: 93.703 ms
     Changes:
              ----------
              diff:
                  New file

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  93.703 ms


# First pkg.refresh_db 
C:\Users\adrian>salt-call --local pkg.refresh_db
local:
    ----------
    failed:
        0
    success:
        0
    total:
        0


# Second pkg.refresh_db 
C:\Users\adrian>salt-call --local pkg.refresh_db
local:
    ----------
    failed:
        0
    success:
        1
    total:
        1

Looks like in the first execution the new .sls found is not found.
Here repeating the same process with full debug logging

C:\Users\adrian>salt-call --local pkg.refresh_db -l debug
C:\Program Files\Salt Project\Salt\bin\lib\site-packages\pkg_resources\__init__.py:123: PkgResourcesDeprecationWarning: p is an invalid version and will not be supported in a future release
  warnings.warn(
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from 'C:\\Program Files\\Salt Project\\Salt\\bin\\Lib\\site-packages\\salt-3004.2-py3.8.egg\\salt\\log\\handlers\\sentry_mod.py'>
[DEBUG   ] Configuration file path: C:\ProgramData\Salt Project\Salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\bin\\Lib\\site-packages\\salt-3004.2-py3.8.egg\\salt\\grains\\zfs.py'>
[DEBUG   ] Loading static grains from C:\ProgramData\Salt Project\Salt\conf\grains
[DEBUG   ] Determining pillar cache
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded pkg.refresh_db
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Refreshing pkg metadata db for saltenv 'base' (age of existing metadata is -1 day, 23:59:59)
[INFO    ] Removing all *.sls files under 'C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[DEBUG   ] LazyLoaded cp.cache_dir
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\adrian, universal_newlines=False, shell=None, istream=None)
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\adrian, universal_newlines=False, shell=None, istream=None)
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[DEBUG   ] Returning file list from cache: age=17 cache_time=20 C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\file_lists\roots\base.p
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded nested.output
local:
    ----------
    failed:
        0
    success:
        0
    total:
        0




C:\Users\adrian>salt-call --local pkg.refresh_db -l debug
C:\Program Files\Salt Project\Salt\bin\lib\site-packages\pkg_resources\__init__.py:123: PkgResourcesDeprecationWarning: p is an invalid version and will not be supported in a future release
  warnings.warn(
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from 'C:\\Program Files\\Salt Project\\Salt\\bin\\Lib\\site-packages\\salt-3004.2-py3.8.egg\\salt\\log\\handlers\\sentry_mod.py'>
[DEBUG   ] Configuration file path: C:\ProgramData\Salt Project\Salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\_schedule.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\autosign.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\module_run.conf
[DEBUG   ] Including configuration from 'C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf'
[DEBUG   ] Reading configuration from C:\ProgramData\Salt Project\Salt\conf\minion.d\performance.conf
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\bin\\Lib\\site-packages\\salt-3004.2-py3.8.egg\\salt\\grains\\zfs.py'>
[DEBUG   ] Loading static grains from C:\ProgramData\Salt Project\Salt\conf\grains
[DEBUG   ] Determining pillar cache
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded pkg.refresh_db
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Refreshing pkg metadata db for saltenv 'base' (age of existing metadata is 0:00:11.846965)
[INFO    ] Removing all *.sls files under 'C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[DEBUG   ] LazyLoaded cp.cache_dir
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\adrian, universal_newlines=False, shell=None, istream=None)
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\adrian, universal_newlines=False, shell=None, istream=None)
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\tanium\test_local.sls
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\tanium\win_repo-nessusagent.sls
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\tanium\win_repo-pkg_refresh.sls
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\tanium\files\Application_Install.json.jinja
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\tanium\files\nessus-agent.sls
[DEBUG   ] Unable to list dir: C:\ProgramData\Salt Project\Salt\srv\salt\win\repo-ng\nessus-agent.sls
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/nessus-agent.sls' to resolve 'salt://win/repo-ng/nessus-agent.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng\nessus-agent.sls' to resolve 'salt://win/repo-ng/nessus-agent.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/nessus-agent.sls'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/nessus-agent.sls'
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng\nessus-agent.sls
[DEBUG   ] Jinja search path: ['C:\\ProgramData\\Salt Project\\Salt\\var\\cache\\salt\\minion\\files\\base']
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[PROFILE ] Time (in seconds) to render 'C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng\nessus-agent.sls' using 'jinja' renderer: 0.0957326889038086
[DEBUG   ] Rendered data from file: C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng\nessus-agent.sls:
nessus-agent:


  '10.3.1.20142':
    full_name: 'Nessus Agent (x64)'
    installer: 'C:\files\NessusAgent-10.3.1-x64.msi'
    install_flags: '/qn /norestart'
    uninstaller: 'C:\files\NessusAgent-10.3.1-x64.msi'
    uninstall_flags: '/qn /norestart'
    msiexec: True
    reboot: False




[DEBUG   ] Results of YAML rendering:
OrderedDict([('nessus-agent', OrderedDict([('10.3.1.20142', OrderedDict([('full_name', 'Nessus Agent (x64)'), ('installer', 'C:\\files\\NessusAgent-10.3.1-x64.msi'), ('install_flags', '/qn /norestart'), ('uninstaller', 'C:\\files\\NessusAgent-10.3.1-x64.msi'), ('uninstall_flags', '/qn /norestart'), ('msiexec', True), ('reboot', False)]))]))])
[PROFILE ] Time (in seconds) to render 'C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\files\base\win\repo-ng\nessus-agent.sls' using 'yaml' renderer: 0.015625953674316406
[DEBUG   ] LazyLoaded nested.output
local:
    ----------
    failed:
        0
    success:
        1
    total:
        1

Setup
WIndows minion 3004.2 in masterless setup
State to put the file in local win/repo-ng folder:

{% set pkg_name = 'nessus-agent' %}

Download_PkgMgr_StateFile:
  file.managed:
    - name: '{{ salt['config.get']('root_dir') }}srv\salt\win\repo-ng\{{ pkg_name }}.sls'
    - source: 'salt://{{ sls_path }}/files/{{ pkg_name }}.sls'
    - skip_verify: True
    - makedirs: True

[amalaguti]

Doing the same on the salt master file system, just adding a file to salt://win/repo-ng, and running salt-call pkg.refresh_db works fine, the new pkg is seen immediately. So seems to be something with --local ?

[twangboy]

#66184 should fix this issue