[BUG] state.apply broken with minion 3007.1 when missing python dependencies

[ptitdoc]

Description
When running salt-master 3007.1, some clients in 3007.1 are not working properly when running state.apply.

I'm trying to understand why since several weeks/months and looking through several bugs related to instabilities with salt 3006.X/3007 which may or may not be related.
#66288
#66282

After looking to running salt-call state.apply directly on the minion because it hangs on the master, I end up with this error:

  File "/usr/lib/python3.12/site-packages/salt/modules/state.py", line 420, in _check_queue
    queue = __salt__["config.option"]("state_queue", False)
            ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/loader/context.py", line 85, in __getitem__
    return self.value()[item]
           ~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/loader/lazy.py", line 350, in __getitem__
    super().__getitem__(item)  # try to get the item from the dictionary
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/utils/lazy.py", line 105, in __getitem__
    raise KeyError(key)
KeyError: 'config.option'

The error comes from module config that cannot loads because it fails to load a bunch of (useless?) dependencies, which seems to be critical as module.config is a core feature:

[ERROR   ] Failed to import module config, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/salt/loader/lazy.py", line 770, in _load_module
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/lib/python3.12/site-packages/salt/modules/config.py", line 22, in <module>
    import salt.utils.cloud
  File "/usr/lib/python3.12/site-packages/salt/utils/cloud.py", line 82, in <module>
    import pkg_resources  # pylint: disable=3rd-party-module-not-gated
    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3291, in <module>
    @_call_aside
     ^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3304, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 600, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 937, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 798, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 839, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'rpm-vercmp' distribution was not found and is required by salt

The problem is traced down to there:

salt/salt/modules/config.py

Line 20 in 246d066

try:

The module properly initialise with HAS_CLOUD set to False if there is an ImportError, so we should not care about the salt.utils.cloud dependencies. But it fails with a DistributionNotFound instead and just crash...

If I remove all the try block and set HAS_CLOUD=False, everything run properly.

Please not that while installing missing dependencies, I also found other exceptions types than ImportError and DistributionNotFound

[dmurphy18]

@ptitdoc FYI - Don't actually have support for Python 3.12 in released code yet (have support under development for 3.11 and 3.12 but not released yet). FreeBSD support has gone to best effort, see https://docs.saltproject.io/salt/install-guide/en/latest/topics/salt-supported-operating-systems.html. But will to take a PR with changes and tests which fixes this on FreeBSD, and the other platforms of course.

In the course of finalizing the next releases for Salt 3006.x and 3007.x, which have a large number of fixes which might resolve some of the issues encountered (esp. in comms on 3007), so the team focus is currently on that, but once the releases are made, perhaps time will be available to check this issue on other platforms, again FreeBSD is best effort.
Sorry, but the team has been suffering from a drastic reduction in size from what it used to be.

[dmurphy18]

@ptitdoc Wondering if this is at all related to the movement of repo.saltproject.io, but that was near the end of October, see #67020

[dmurphy18]

@ptitdoc Closing this due to unresponsiveness. There will soon be a release Salt 3006.10 and 3007.2, please retry once either of these version are released, and feel free to reopen this issue with additional details.

Noting, that Salt is tested with Python 3.10 and does have problems with Python 3.11, 3.12 and 3.13, these versions of Python are being addressed but the code is not ready for prime-time yet.