SaltSSH Bootstrap Error on Ubuntu 20.04 Client

[bezkarl]

Problem description

When trying to bootstrap a Ubuntu 20.04 client from the Uyuni Web UI, the boostrap fails with the error:

"ERROR com.suse.manager.webui.controllers.utils.AbstractMinionBootstrapper - Error during bootstrap: SaltSSHError(13, stderr: "", stdout: "ERROR: Failure deploying ext_mods:"

Version of Uyuni Server and Proxy (if used)

2022.05-180.1.uyuni1, no proxy

Details about the issue

Hi,

I am attempting to bootstrap from the Uyuni Web UI using a user created on client/minion that is a member of the sudo group, with NOPASSWD set using visudo.

The minion bootstrap fails with the error above, which is found in the /var/log/rhn_web_ui.log

I can "onboard" the minion successfully using a bootstrap script run as root on the minion itself, so it seems this a permissions error when trying through SaltSSH.

I have noticed that a file created as part of the SaltSSH bootstrap in /var/tmp is owned by root and not the user running the bootstrap:

drwxr-xr-x 3 root root 4096 May 31 10:16 .salt_a89cdf_salt

I have tried changing the ownership of this file to the user running the bootstrap and re-run the process. This bypasses the above error and continues to install the venv-salt-minion and start the service. However the minion never appears in the Web UI even though the Salt key shows as accepted, so I am not sure whether changing the file permissions causes another issue which I haven't fully investigated.

While I can workound the issue and bootstrap by running the script on the minion, this may become problematic as we widen the scope and use of our Uyuni instance.

Please let me know if there are any log files I can provide to assist troubleshooting.

Kind regards,
Karl

[aaannz]

I am assuming that ssh_push_sudo_user option is correctly set in /etc/rhn/rhn.conf to the user you want to use on the minion.

Can you upload complete /var/log/rhn_web_ui.log as there seems to be 2 issues. One with the mentioned ERROR: Failure deploying ext_mods:" and then another one after salt key is accepted registration for some reason fails.

[bezkarl]

Hi,

Many thanks for the response.

ssh_push_sudo_user wasn't set but I have done this and re-tried but it made no difference. (I did restart the salt-manager service after the change assuming this would consume the config).

I have uploaded the log file /var/log/rhn_web_ui.log but apart from the odd auth failure due to me typing a password incorrectly there doesn't seem anything obvious- is it possible I can turn up the logging level to assist further?

Interestingly I was able to run a remote command on the minion using mgr-salt-ssh after cleaning any minion files, and the /var/tmp file permissions were different from those used by the Web UI bootstrap.

Kind regards,
Karl

rhn_web_ui.log

[santeri3700]

I'm also experiencing this issue with AlmaLinux 8 and CentOS 7 clients. Bootstrapping via Bootstrap Script works, but not via Web UI without a workaround.
I'm attempting to bootstrap a system which is managed via Salt Minion so I have unchecked the "Manage system completely via SSH (will not install an agent)" option. Bootstrapping user is either "almalinux" or "centos" which both have passwordless sudo allowed. This has worked in the past (2022.03).

Uyuni Server 2022.05 (upgraded from previous releases since 2022.01). Tested clients are clean installed systems.

Uyuni Server Web UI error:

SaltSSHError(13, stderr: "", stdout: "ERROR: Failure deploying ext_mods: _edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878 ext_mods")

And the same in rhn_web_ui.log. There doesn't seem to be anything else related other than this line:

2022-06-06 13:18:43,383 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] ERROR com.suse.manager.webui.controllers.utils.AbstractMinionBootstrapper - Error during bootstrap: SaltSSHError(13, stderr: "", stdout: "ERROR: Failure deploying ext_mods: _edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878 ext_mods")

The ext_mods seem to be related to "Salt Thin" or something? I do not have web.ssh_use_salt_thin enabled in rhn.conf in case you're wondering.

CentOS 7 client:

Uyuni Server seems to be able to push some of the files, but installation fails due to incorrect permissions. The system will not appear in Uyuni even if the permissions are fixed after first failing a bootstrap attempt and then re-attempting another time.

[centos@test ~]$ ls -l /var/tmp/venv-salt-minion/
total 36
drwxr-xr-x. 3 centos centos   265  6.6. 10:18 bin
drwxr-xr-x. 3 centos centos   142  6.6. 10:18 lib
lrwxrwxrwx. 1 centos centos     3  6.6. 10:18 lib64 -> lib
-rw-r--r--. 1 centos centos 26062 13.4. 11:30 packages.log
-rw-r--r--. 1 centos centos    64 13.4. 11:30 pyvenv.cfg
drwxr-xr-x. 2 centos centos    41  6.6. 10:18 usr
-rw-rw-r--. 1 centos centos    65  6.6. 10:18 venv-hash.txt

[centos@test ~]$ ls -la /var/tmp/.centos_58bcca_salt/
total 4
drwxr-xr-x. 3 root root  40  6.6. 10:18 .
drwxrwxrwt. 6 root root 154  6.6. 10:55 ..
-rw-r--r--. 1 root root 208  6.6. 10:55 minion
drwxr-xr-x. 3 root root  17  6.6. 10:18 running_data

Workaround

Theoretically this issue shouldn't happen when bootstrapping with the root user, but it's usually not allowed to SSH in for security reasons.
You can either try bootstrapping with root user or use the workaround below.

1. Delete all temporary files created on the client by previous failed attempts

$ sudo rm -rf /var/tmp/venv-salt-minion
$ sudo rm -rf /var/tmp/.*_salt/

2. Run a while loop on the client which constantly attempts to fix the ownership of the temporary files. This assumes you're logged in as the bootstrap user.

$ while true; do sudo chown -R $USER:$USER /var/tmp/.*_salt; sleep 0.1; done
(You will see some errors which can be ignored)

3. Wait for the bootstrapping to complete and see if the system is listed in the Salt Keys and Systems. Exit the while loop by pressing Ctrl+C.

[santeri3700]

And to confirm my theory, bootstrapping via Web UI as the root user does indeed work without any workarounds etc.. My workaround isn't necessarily needed if you can (temporarily) allow root ssh login.

[bezkarl]

Thanks for the while loop workround @santeri3700.

I did find that just changing the ownership after the first bootstrap run of the /var/temp/.*_salt directory to uyuni user and also chmod 755, did allow me to complete the bootstrap on the second run.

I identified that the reason that clients weren't showing in the UI after changing the temp directory ownership/chmod was that being cloned VMs the machine ID was the same and needed changing.

[vzhestkov]

@santeri3700 thanks for pointing the workaound, it helps to find the solution.

[vzhestkov]

I've reproduced the issue, preparing the fix for it.

[bezkarl]

Thanks very much @vzhestkov

[vzhestkov]

@bezkarl actually after failing you can run chown -R $USER:$USER /var/tmp/.*_salt once and make one more attempt to bootstrap, the second run should work. Sure it's also workaround and quite ugly.
Will try to promote the proper fix as soon as possible.

[bezkarl]

@vzhestkov I ran the above command after the the inital failure as the uyuni user, and the second bootstrap was successful.

[vzhestkov]

The fix was merged and promoted: openSUSE/salt#534
But it will take some time to get the fix published for openSUSE as salt package is promoted there from SLE. The fix will be available with 2022.09.

[bezkarl]

Thanks @vzhestkov

[mcalmer]

I think this issue can be closed