[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

ggiesen · 2023-10-20T14:41:18Z

Description
Sometime around 3006.0 - 3006.2 I started getting errors on my mount.mounted states for NFS mounts:

----------
          ID: mount_/etc/salt/pki/master/minions
    Function: mount.mounted
        Name: /etc/salt/pki/master/minions
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2381, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1293, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/mount.py", line 242, in mounted
                  active = __salt__["mount.active"](extended=True)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/mount.py", line 277, in active
                  _active_mountinfo(ret)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/mount.py", line 59, in _active_mountinfo
                  __context__["disk.blkid"] = __salt__["disk.blkid"]()
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/context.py", line 78, in __getitem__
                  return self.value()[item]
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 349, in __getitem__
                  super().__getitem__(item)  # try to get the item from the dictionary
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/lazy.py", line 105, in __getitem__
                  raise KeyError(key)
              KeyError: 'disk.blkid'
     Started: 14:32:19.750162
    Duration: 34.566 ms
     Changes:

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

on-prem machine
VM (Virtualbox, KVM, etc. please specify)
VM running on a cloud service, please be explicit and add details
container (Kubernetes, Docker, containerd, etc. please specify)
or a combination, please be explicit
jails if it is FreeBSD
classic packaging
onedir packaging
used bootstrap to install

State:

  mount_/etc/salt/pki/master/minions:
    mount.mounted:
    - name: /etc/salt/pki/master/minions
    - device: 192.0.2.97:/etc/salt/pki/master/minions
    - fstype: nfs4
    - opts:
      - defaults
      - rw
      - nosuid
      - nodev
      - noexec
      - relatime
      - vers=4.1
      - rsize=65536
      - wsize=65536
      - namlen=255
      - hard
      - proto=tcp
      - timeo=600
      - retrans=2
      - sec=sys
      - local_lock=none
    - persist: true
    - mount: true

Steps to Reproduce the behavior
Try to apply the above state

Expected behavior
State would report no changes

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
              Salt: 3006.3

Python Version:
            Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
              cffi: 1.14.6
          cherrypy: unknown
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 3.1.2
           libgit2: 1.5.0
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.9.8
            pygit2: 1.11.1
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 23.2.0
            relenv: 0.13.10
             smmap: Not Installed
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4

Salt Extensions:
 saltext.bitwarden: 0.0.1b15
    saltext.vmware: 23.6.29.0rc1

System Versions:
              dist: almalinux 8.7 Stone Smilodon
            locale: utf-8
           machine: x86_64
           release: 4.18.0-425.19.2.el8_7.x86_64
            system: Linux
           version: AlmaLinux 8.7 Stone Smilodon

The text was updated successfully, but these errors were encountered:

ggiesen · 2023-10-20T15:08:43Z

My guess it it was broken by @nicholasmhughes PR here: #63104 which looks like it was merged for 3006.0

nicholasmhughes · 2023-10-20T15:11:29Z

stop pointing fingers @ggiesen 😉

ggiesen · 2023-10-20T15:11:59Z

lol I just thought you might be the best guy to fix it ;)

ggiesen · 2023-10-20T15:13:13Z

FYI NFS mounts don't show up with the blkid command:

# blkid
/dev/sda1: UUID="8256a0b9-70ef-4798-bf83-58859b5702fc" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="3ba51781-01"
/dev/sda2: UUID="yVSzYj-cniL-R7KW-OwSq-bsQE-lHTz-s1NayN" TYPE="LVM2_member" PARTUUID="3ba51781-02"
/dev/mapper/almalinux-root: UUID="9d9ae71e-473b-4c61-b67c-29bd242c7fb1" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/almalinux-swap: UUID="a23fc7b6-590a-4925-a707-491c6edd78dc" TYPE="swap"
/dev/mapper/almalinux-tmp: UUID="2ee59e1c-2551-4807-8e97-2bca72d39d0f" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/almalinux-home: UUID="5b72135d-401d-4f05-8cab-882b24dc8604" BLOCK_SIZE="512" TYPE="xfs

nicholasmhughes · 2023-10-20T15:19:00Z

I don't see anywhere in that PR of mine where I changed disk.blkid itself. The error looks like the LazyLoader can't find that function, which usually happens when there's a syntax error in the execution module somewhere. Can you run disk.blkid by itself? Are you by chance overriding any execution modules from the Salt filesystem?

ggiesen · 2023-10-20T15:24:32Z

Very astute, you are correct in that disk.blkid is not working:

[DEBUG   ] Could not LazyLoad disk.blkid: 'disk.blkid' is not available.
'disk.blkid' is not available.

I'm not, as far as I'm aware, I'm not overriding any execution modules from Salt (unless perhaps the saltext.vmware extension is, but that wouldn't touch the disk module). I'm currently dealing with more relenv packaging disasters through so who the heck knows.

nicholasmhughes · 2023-10-20T15:25:59Z

Do any of the other disk functions work? What about disk.usage?

ggiesen · 2023-10-20T15:26:36Z

Nope.

nicholasmhughes · 2023-10-20T15:30:51Z

There's definitely something borked on the system(s). The disk module is either missing or corrupt, or alternatively, Python path issues are preventing some modules from being found. You mentioned relenv issues... are you packaging Salt yourself?

ggiesen · 2023-10-20T15:30:54Z

I think I found it:

# salt-call disk.blkid -l trace
...
[TRACE   ] Loaded disk as virtual vmware_disk
[DEBUG   ] The functions from module 'vmware_disk' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'vmc_vm_disks' are being loaded by dir() on the loaded module
...
[DEBUG   ] Could not LazyLoad disk.blkid: 'disk.blkid' is not available.
'disk.blkid' is not available.

# salt-pip uninstall saltext.vmware
Found existing installation: saltext.vmware 23.6.29.0rc1
Uninstalling saltext.vmware-23.6.29.0rc1:
  Would remove:
    /opt/saltstack/salt/extras-3.10/saltext.vmware-23.6.29.0rc1.dist-info/*
    /opt/saltstack/salt/extras-3.10/saltext/vmware/*
Proceed (Y/n)? y
  Successfully uninstalled saltext.vmware-23.6.29.0rc1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv


# salt-call disk.blkid
local:
    ----------
    /dev/mapper/almalinux-home:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            5b72135d-401d-4f05-8cab-882b24dc8604
    /dev/mapper/almalinux-root:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            9d9ae71e-473b-4c61-b67c-29bd242c7fb1
    /dev/mapper/almalinux-swap:
        ----------
        TYPE:
            swap
        UUID:
            a23fc7b6-590a-4925-a707-491c6edd78dc
    /dev/mapper/almalinux-tmp:
        ----------
        BLOCK_SIZE:
            512
        TYPE:
            xfs
        UUID:
            2ee59e1c-2551-4807-8e97-2bca72d39d0f
    /dev/sda1:
        ----------
        BLOCK_SIZE:
            512
        PARTUUID:
            3ba51781-01
        TYPE:
            xfs
        UUID:
            8256a0b9-70ef-4798-bf83-58859b5702fc
    /dev/sda2:
        ----------
        PARTUUID:
            3ba51781-02
        TYPE:
            LVM2_member
        UUID:
            yVSzYj-cniL-R7KW-OwSq-bsQE-lHTz-s1NayN

nicholasmhughes · 2023-10-20T15:37:23Z

Well... that's terrible.

ggiesen · 2023-10-20T15:38:20Z

Yep, I really was hoping it was your fault because there'd be a much higher chance of being fixed in a non-galactic timeframe.

ggiesen added Bug broken, incorrect, or confusing behavior needs-triage labels Oct 20, 2023

ggiesen mentioned this issue Oct 20, 2023

Extension conflicts with disk module in Salt saltstack/salt-ext-modules-vmware#388

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

ggiesen commented Oct 20, 2023 •

edited

Loading

ggiesen commented Oct 20, 2023 •

edited

Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023

ggiesen commented Oct 20, 2023

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 •

edited

Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 •

edited

Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 •

edited

Loading

[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

[BUG] KeyError: 'disk.blkid' on mount.mounted for NFS mount #65441

Comments

ggiesen commented Oct 20, 2023 • edited Loading

ggiesen commented Oct 20, 2023 • edited Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023

ggiesen commented Oct 20, 2023

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 • edited Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 • edited Loading

nicholasmhughes commented Oct 20, 2023

ggiesen commented Oct 20, 2023 • edited Loading

ggiesen commented Oct 20, 2023 •

edited

Loading

ggiesen commented Oct 20, 2023 •

edited

Loading

ggiesen commented Oct 20, 2023 •

edited

Loading

ggiesen commented Oct 20, 2023 •

edited

Loading

ggiesen commented Oct 20, 2023 •

edited

Loading