[BUG] salt minion 3006.5 fails to work on customized Ubuntu 20.04.6

[mato22222]

Description
After installing salt-minion 3006.5 I am receiving this error when running state file:
salt-minion[7955]: [ERROR ] State 'pkg.installed' was not found in SLS '_states.install-api-prereq'
salt-minion[7955]: Reason: 'pkg' virtual returned False: pkg module could not be loaded
[salt.state :323 ][ERROR ][7955] State 'pkg.installed' was not found in SLS '_states.install-api-prereq'#012Reason: 'pkg' virtual returned False: pkg module could not be loaded

Setup
install_prereq.sls

install_dos2unix:
pkg.installed:
- name: dos2unix
install_python3_filelock:
pkg.installed:
- name: python3-filelock

State file is triggered by reactor, but same result was obtained by running salt state.apply

Please be as specific as possible and give set-up details.

• on-prem machine
• [x ] VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• [x ] onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
install Ubuntu 20.04 - customized
install salt 3006.5
execute state file
(it works with 3006.4 and older. But after update to 3006.5 it fails as above.
Also works with 3005.x)

(Include debug logs if possible and relevant)

Expected behavior
pkg.installed should install ubuntu package

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

Master: Salt Version: Salt: 3006.5

Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: ubuntu 20.04.6 focal
locale: utf-8
machine: x86_64
release: 5.4.0-169-generic
system: Linux
version: Ubuntu 20.04.6 focal

Minion:

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt-minion --versions-report
Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: 0.40.1
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.15.0-91-generic
        system: Linux
       version: Custom OS 20.04.6 focal

Additional context
Minion is customized Ubuntu:
real name was replaced with Custom in output

cat /etc/os-release
NAME="Custom OS"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Custom OS"
VERSION_ID="20.04"
HOME_URL="https://www..t-systems.com/"
SUPPORT_URL="https://help..t-systems.com/"
PRIVACY_POLICY_URL="https://www..t-systems.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Add any other context about the problem here.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[lkubb]

Pretty sure the aptpkg execution module does not load because of the custom OS name. You can check if grains.get os_family returns Debian - if not, that's the cause.

To fix, see https://docs.saltproject.io/en/latest/ref/modules/index.html#overriding-virtual-module-providers

Edit: As to why it started failing in 3006.5, I suspect there is a custom grain module that overrides at least osfamily + #65692.

[mato22222]

There is no custom grain module. We are currently not working with grains at all (except reactor, but this is read only).
Seems salt grain output has changed between 3006.4 and 3006.5. This points to different fields of os-release
NAME or PRETTT_NAME instead of ID or ID_LIKE or even different source.

3006.5 output
salt '9864-fdbd775b9fdf.local' grains.get os_family
9864-fdbd775b9fdf.local:
Custom OS

3006.4 output
salt '9864-fdbd775b9fdf.local' grains.get os_family
9864-fdbd775b9fdf.local:
Debian

[Akm0d]

Related to #59061

[Akm0d]

As a quick fix you could run:

salt "9864-fdbd775b9fdf.local" grains.set os_family Debian
salt "9864-fdbd775b9fdf.local" saltutil.refresh_grains

[Akm0d]

@mato22222 , this may be related to a bug introduced with #65027, which was reverted in salt 3006.7. Can you upgrade to see if the issue persists on that version?

[mato22222]

by setting grains, issue is solved. Did not have time to test with latest version yet.