Update from 16.03 to 16.09, problems with /var/setuid-wrappers #19862

bsoudan · 2016-10-25T15:37:53Z

Issue description

I just ran an update from 16.03 to 16.09, and at the end of the update, I noticed this:

umount: /var/setuid-wrappers: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
rm: cannot remove '/var/setuid-wrappers': Device or resource busy

Then, in another window, tried a sudo operation but received:

sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

I poked around a bit, and /var/setuid-wrappers looked a bit wacky, it looked like there were too many levels, as if a symlink operation went bad:

[root@bsoudan-linux:/home/bsoudan]# cd /var/setuid-wrappers/

[root@bsoudan-linux:/var/setuid-wrappers]# ls -al
total 4
drwxr-xr-x 2 root root   60 Oct 25 11:22 .
drwxr-xr-x 8 root root 4096 Oct 10 10:25 ..
lrwxrwxrwx 1 root root   51 Oct 25 11:22 setuid-wrappers.AvZKKe7ojo -> /run/setuid-wrapper-dirs/setuid-wrappers.AvZKKe7ojo

[root@bsoudan-linux:/var/setuid-wrappers]# cd setuid-wrappers.AvZKKe7ojo/

[root@bsoudan-linux:/var/setuid-wrappers/setuid-wrappers.AvZKKe7ojo]# ls -al
total 260
drwxr-xr-x 2 root root         560 Oct 25 11:22 .
drwxr-xr-x 3 root root          60 Oct 25 11:22 ..
-r-s--x--x 1 root root       12856 Oct 25 11:22 chfn
-rw-r--r-- 1 root root          65 Oct 25 11:22 chfn.real
-r-sr-x--- 1 root messagebus 12856 Oct 25 11:22 dbus-daemon-launch-helper
-rw-r--r-- 1 root root          90 Oct 25 11:22 dbus-daemon-launch-helper.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 fusermount
-rw-r--r-- 1 root root          69 Oct 25 11:22 fusermount.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 newgidmap
-rw-r--r-- 1 root root          70 Oct 25 11:22 newgidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 newuidmap
-rw-r--r-- 1 root root          70 Oct 25 11:22 newuidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 ping
-r-s--x--x 1 root root       12856 Oct 25 11:22 ping6
-rw-r--r-- 1 root root          70 Oct 25 11:22 ping6.real
-rw-r--r-- 1 root root          69 Oct 25 11:22 ping.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 pkexec
-rw-r--r-- 1 root root          71 Oct 25 11:22 pkexec.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 polkit-agent-helper-1
-rw-r--r-- 1 root root          91 Oct 25 11:22 polkit-agent-helper-1.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 su
-r-s--x--x 1 root root       12856 Oct 25 11:22 sudo
-r-s--x--x 1 root root       12856 Oct 25 11:22 sudoedit
-rw-r--r-- 1 root root          66 Oct 25 11:22 sudoedit.real
-rw-r--r-- 1 root root          66 Oct 25 11:22 sudo.real
-rw-r--r-- 1 root root          66 Oct 25 11:22 su.real
-r-s--x--x 1 root nogroup    12856 Oct 25 11:22 unix_chkpwd
-rw-r--r-- 1 root root          81 Oct 25 11:22 unix_chkpwd.real

I tried running another 'nixos-rebuild switch' operation, and that appeared to fix everything. No errors this time:

[root@bsoudan-linux:/run/setuid-wrapper-dirs/setuid-wrappers.AvZKKe7ojo]# nixos-rebuild switch
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...

[bsoudan@bsoudan-linux:~]$ cd /var/setuid-wrappers

[bsoudan@bsoudan-linux:/var/setuid-wrappers]$ ls -al
total 260
drwxr-xr-x 2 root root         560 Oct 25 11:26 .
drwxr-xr-x 4 root root          80 Oct 25 11:26 ..
-r-s--x--x 1 root root       12856 Oct 25 11:26 chfn
-rw-r--r-- 1 root root          65 Oct 25 11:26 chfn.real
-r-sr-x--- 1 root messagebus 12856 Oct 25 11:26 dbus-daemon-launch-helper
-rw-r--r-- 1 root root          90 Oct 25 11:26 dbus-daemon-launch-helper.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 fusermount
-rw-r--r-- 1 root root          69 Oct 25 11:26 fusermount.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 newgidmap
-rw-r--r-- 1 root root          70 Oct 25 11:26 newgidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 newuidmap
-rw-r--r-- 1 root root          70 Oct 25 11:26 newuidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 ping
-r-s--x--x 1 root root       12856 Oct 25 11:26 ping6
-rw-r--r-- 1 root root          70 Oct 25 11:26 ping6.real
-rw-r--r-- 1 root root          69 Oct 25 11:26 ping.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 pkexec
-rw-r--r-- 1 root root          71 Oct 25 11:26 pkexec.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 polkit-agent-helper-1
-rw-r--r-- 1 root root          91 Oct 25 11:26 polkit-agent-helper-1.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 su
-r-s--x--x 1 root root       12856 Oct 25 11:26 sudo
-r-s--x--x 1 root root       12856 Oct 25 11:26 sudoedit
-rw-r--r-- 1 root root          66 Oct 25 11:26 sudoedit.real
-rw-r--r-- 1 root root          66 Oct 25 11:26 sudo.real
-rw-r--r-- 1 root root          66 Oct 25 11:26 su.real
-r-s--x--x 1 root nogroup    12856 Oct 25 11:26 unix_chkpwd
-rw-r--r-- 1 root root          81 Oct 25 11:26 unix_chkpwd.real

I'm worried though if I would have rebooted my system would have been foobar'd, even if I booted into an old system.

System:

[bsoudan@bsoudan-linux:~]$ /run/booted-system/sw/bin/nixos-version 
16.03.1242.c5cbda2 (Emu)

[bsoudan@bsoudan-linux:~]$ /run/current-system/sw/bin/nixos-version 
16.09.824.e4fb65a (Flounder)

Nix version:

[bsoudan@bsoudan-linux:~]$ /run/current-system/sw/bin/nix-env --version
nix-env (Nix) 1.11.4

[bsoudan@bsoudan-linux:~]$ /run/booted-system/sw/bin/nix-env --version
nix-env (Nix) 1.11.2

Nixpkgs version: "16.09.824.e4fb65a

The text was updated successfully, but these errors were encountered:

bsoudan · 2016-10-25T15:46:52Z

Related to #18186 and #18124 probably?

FRidh · 2016-11-04T10:19:37Z

cc @domenkozar

domenkozar · 2016-11-04T10:22:33Z

Did you upgrade to 16.09 with sudo?

bsoudan · 2016-11-04T20:19:36Z

Probably, but I couldn't say for sure, the upgrade is definitely long out of my bash history. :(

res0nat0r · 2017-03-01T19:22:26Z

I appear to be getting hit by this.

$ sudo su -
sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

$ ls -l /run/current-system/sw/bin/sudo
lrwxrwxrwx 1 root root 66 Jan  1  1970 /run/current-system/sw/bin/sudo -> /nix/store/sgw7jnji7fclpfw7kn049ilrnsrn06x5-sudo-1.8.19p2/bin/sudo

$ ls -l /run/current-system/sw/bin/sudo -L
-r-xr-xr-x 1 root root 149064 Jan  1  1970 /run/current-system/sw/bin/sudo

$  /run/booted-system/sw/bin/nixos-version
16.09.1512.6b28bd0 (Flounder)

$ /run/current-system/sw/bin/nixos-version
17.03pre101839.53a2baa (Gorilla)

I did run nixos-rebuild switch via sudo. I'm not sure how to salvage this now that I'm not able to get root access as I don't have a manual root passwd currently set.

domenkozar · 2017-03-01T19:31:07Z

Do you still have logs for the upgrade?

res0nat0r · 2017-03-01T19:45:55Z

@domenkozar I haven't rebooted the host, is there a specific logfile where this is stored that I could attach for you?

domenkozar · 2017-03-01T19:51:09Z

It's just in the standard output. My hypothesis is, since you use sudo it's not able to run the migration for the wrappers.

res0nat0r · 2017-03-01T19:54:09Z

@domenkozar Ah, sorry unfortunately then I do not. I was adding some NFS mounts to my configuration.nix and restarting the network services killed my ssh session so I don't have the upgrade log anymore.

If I look into booting into single user mode and chmod u+s to the sudo binary would that be enough to fix this issue or do you think there are bigger issues still lingering?

res0nat0r · 2017-03-01T20:23:40Z

Actually now I see that the sudo binary does in fact have the correct permissions. I logged out of the host and logged in again and am now able to sudo su - to root.

$ ls -lL `which sudo`
-r-s--x--x 1 root root 17728 Mar  1 19:11 /run/wrappers/bin/sudo

Could it be that my existing shell which ran the sudo nixos-rebuild switch just wasn't properly pointing to the new sudo binary due to some shell env swap breakage and that the upgrade in fact worked as intended?

domenkozar · 2017-03-01T20:29:34Z

Ah yes, it probably points to the old /run/setuid-wrappers/sudo.

qknight · 2017-03-02T18:09:56Z

i've upgraded from 16.09 to master and i couldn't use su anymore:

➜  ~ su
Password: 
su: Authentication failure
➜  ~ which su   
/run/current-system/sw/bin/su

but running a different su tool:

➜  ~ /run/wrappers/bin/su 
Password: 
root@lenovo-t530>

using the other binary it is working again as a quick solution!

thanks @domenkozar

Mic92 · 2017-03-02T18:54:28Z

@qknight you logout and login again to get /run/wrappers/bin/ in your PATH.

teh · 2017-03-08T10:19:30Z

Minimal test case for a fresh 16.09 system:

sudo nano /root/.nix-channels # update to 17.03
sudo nix-channel --update
sudo nixos-rebuild switch

[tom@test:~]$ sudo
sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

@ixmatus cc'ing you in case you have a simple fix in mind, otherwise I'll give it a shot.

qknight · 2017-03-08T10:22:21Z

@teh: simply 'source /etc/profile' and your commands should be working.

are you using zsh?

teh · 2017-03-08T10:29:17Z

@qknight Using bash. To be clear: I added a test case so we can fix. I also tested running as root wich works as intended. @domenkozar's theory:

It's just in the standard output. My hypothesis is, since you use sudo it's not able to run the migration for the wrappers.

seems to be correct - the combination of sudo and switch breaks

ixmatus · 2017-03-08T11:07:19Z

The paths change due to the switch to using /run/wrappers/bin so you need to log back in to pick up the PATH changes. Does that fix the issue? How do we communicate this in a upgrade?

domenkozar · 2017-03-08T11:08:19Z

It would be easiest just to not switch paths. It's more pain than gain On Wed, Mar 8, 2017, 12:07 Parnell Springmeyer <notifications@github.com> wrote:

…

The paths change due to the switch to using /run/wrappers/bin so you need to log back in to pick up the PATH changes. Does that fix the issue? How do we communicate this in a upgrade? On Wed, Mar 8, 2017, 4:29 AM teh ***@***.***> wrote: > @qknight <https://github.com/qknight> Using bash. To be clear: I added a > test case so we can fix. I also tested running as root wich works as > intended. @domenkozar <https://github.com/domenkozar>'s theory: > > It's just in the standard output. My hypothesis is, since you use sudo > it's not able to run the migration for the wrappers. > > seems to be correct - the combination of sudo and switch breaks > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#19862 (comment)>, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAB3-gLP1UoKgk2rTDQ3p7rENFJ-LmMAks5rjoMAgaJpZM4KgJEG > > . > -- Parnell Springmeyer 0xDCCF89258EAD874A <http://pgp.mit.edu/pks/lookup?op=get&search=0xDCCF89258EAD874A> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19862 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHtg_fEYW8pTD3W7jMK2JOJeXk4wRluks5rjovogaJpZM4KgJEG> .

ixmatus · 2017-03-08T11:16:05Z

I originally had kept it that way but was asked to remove the indirection if the var path.

ixmatus · 2017-03-08T11:19:19Z

Well that's not entirely true, actually, my original changes even with the var path would have broken this but it was in an attempt to unify all of this name things accurately.

ixmatus · 2017-03-08T11:20:56Z

What if we have a symlink of the old setuid path to this new path??

ixmatus · 2017-03-08T11:23:25Z

Also, I'd like to provide the fix for this since I should have thought about the migration path more thoroughly. I can do so in about two hours.

teh · 2017-03-08T11:44:54Z

I'm actually surprised that the combination of sudo and switch doesn't break more often. I guess it's because the PATH doesn't change a lot.

But generally sudo spawns a new shell and on exit it loses the environment changes. I'm not 100% clear that a symlink is the best solution as it's going to hang around after when its no longer needed.

teh · 2017-03-08T12:03:02Z

The more I think about this the I'm leaning towards adding an entry to the release notes for now. re-sourcing /etc/profile or logging out and in again doesn't seem like an undue burden after an os upgrade with sudo. Still suboptimal though!

In general it'd be useful to have a mechanism that re-sources the environment after a switch, no matter how that switch was executed. direnv solves that by running a no-output function when rendering the prompt, but it seems heavy to opt-in every user to a mechanism like that.

For this specific case we also have a chicken-and-egg problem, because we can't deliver a fix that will arrive in users' open shells before they do the os upgrade.

ixmatus · 2017-03-08T14:18:52Z

I'm available to discuss and work on this. Personally, I don't want to disrupt people's workflows too much but I also fall on the side of just getting this over with since I think it needs to happen.

A symlink pointing to the new path isn't too dirty and can be removed after the 17.03 release for the 17.09 release, I think that's fairly reasonable.

domenkozar · 2017-03-08T14:20:03Z

I'm skeptical symlink will work as wrappers assert the path they're being ran from (at least that was the case before the refactoring).

ixmatus · 2017-03-08T14:48:11Z

@domenkozar shoot, you're right.

ixmatus · 2017-03-08T14:54:19Z

In any case, I think rm'ing the /var/setuid-wrappers was a bad move on my part not thinking about running from sudo itself; I will put up a PR to eliminate that step which should fix some of the issues people have experienced as reported in this issue.

ixmatus · 2017-03-08T15:03:39Z

@domenkozar @teh we can move discussion of how to handle the migration over to #23641. I'm at-minimum removing the code that deletes the old wrapper dirs as that will significantly reduce migration pain I think.

I'm testing on my EC2 machine from scratch with some of the actions people reported here.

qknight · 2017-03-20T23:10:04Z

@domenkozar
i had been using zsh with oh-my-zsh and this hardcoded the PATH into the shell script. removing that variable made it work!

thanks for all the help!

globin · 2017-03-22T11:15:31Z

The 17.03 stuff is handled in #23641

This makes setuid wrappers not fail after upgrading. references #23641, #22914, #19862, #16654

This makes setuid wrappers not fail after upgrading. references #23641, #22914, #19862, #16654 (cherry picked from commit e82baf0)

This makes setuid wrappers not fail after upgrading. references NixOS#23641, NixOS#22914, NixOS#19862, NixOS#16654 (cherry picked from commit e82baf0)

NeQuissimus added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 9.needs: community feedback labels Oct 25, 2016

domenkozar added this to the 17.03 milestone Mar 1, 2017

ixmatus mentioned this issue Mar 8, 2017

security-wrapper: Don't remove the old paths yet as that can create migration pain #23641

Merged

7 tasks

globin closed this as completed Mar 22, 2017

globin added a commit that referenced this issue Mar 23, 2017

security-wrapper: link old wrapper dir to new one

e82baf0

This makes setuid wrappers not fail after upgrading. references #23641, #22914, #19862, #16654

globin added a commit that referenced this issue Mar 23, 2017

security-wrapper: link old wrapper dir to new one

1394a0c

This makes setuid wrappers not fail after upgrading. references #23641, #22914, #19862, #16654 (cherry picked from commit e82baf0)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update from 16.03 to 16.09, problems with /var/setuid-wrappers #19862

Update from 16.03 to 16.09, problems with /var/setuid-wrappers #19862

bsoudan commented Oct 25, 2016

bsoudan commented Oct 25, 2016

FRidh commented Nov 4, 2016

domenkozar commented Nov 4, 2016

bsoudan commented Nov 4, 2016

res0nat0r commented Mar 1, 2017 •

edited

Loading

domenkozar commented Mar 1, 2017

res0nat0r commented Mar 1, 2017

domenkozar commented Mar 1, 2017

res0nat0r commented Mar 1, 2017 •

edited

Loading

res0nat0r commented Mar 1, 2017 •

edited

Loading

domenkozar commented Mar 1, 2017

qknight commented Mar 2, 2017

Mic92 commented Mar 2, 2017 •

edited

Loading

teh commented Mar 8, 2017

qknight commented Mar 8, 2017

teh commented Mar 8, 2017

ixmatus commented Mar 8, 2017 via email •

edited

Loading

domenkozar commented Mar 8, 2017 via email

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

teh commented Mar 8, 2017

teh commented Mar 8, 2017

ixmatus commented Mar 8, 2017

domenkozar commented Mar 8, 2017

ixmatus commented Mar 8, 2017

ixmatus commented Mar 8, 2017

ixmatus commented Mar 8, 2017

qknight commented Mar 20, 2017

globin commented Mar 22, 2017

Update from 16.03 to 16.09, problems with /var/setuid-wrappers #19862

Update from 16.03 to 16.09, problems with /var/setuid-wrappers #19862

Comments

bsoudan commented Oct 25, 2016

Issue description

bsoudan commented Oct 25, 2016

FRidh commented Nov 4, 2016

domenkozar commented Nov 4, 2016

bsoudan commented Nov 4, 2016

res0nat0r commented Mar 1, 2017 • edited Loading

domenkozar commented Mar 1, 2017

res0nat0r commented Mar 1, 2017

domenkozar commented Mar 1, 2017

res0nat0r commented Mar 1, 2017 • edited Loading

res0nat0r commented Mar 1, 2017 • edited Loading

domenkozar commented Mar 1, 2017

qknight commented Mar 2, 2017

Mic92 commented Mar 2, 2017 • edited Loading

teh commented Mar 8, 2017

qknight commented Mar 8, 2017

teh commented Mar 8, 2017

ixmatus commented Mar 8, 2017 via email • edited Loading

domenkozar commented Mar 8, 2017 via email

ixmatus commented Mar 8, 2017 via email • edited Loading

ixmatus commented Mar 8, 2017 via email • edited Loading

ixmatus commented Mar 8, 2017 via email • edited Loading

ixmatus commented Mar 8, 2017 via email • edited Loading

teh commented Mar 8, 2017

teh commented Mar 8, 2017

ixmatus commented Mar 8, 2017

domenkozar commented Mar 8, 2017

ixmatus commented Mar 8, 2017

ixmatus commented Mar 8, 2017

ixmatus commented Mar 8, 2017

qknight commented Mar 20, 2017

globin commented Mar 22, 2017

res0nat0r commented Mar 1, 2017 •

edited

Loading

res0nat0r commented Mar 1, 2017 •

edited

Loading

res0nat0r commented Mar 1, 2017 •

edited

Loading

Mic92 commented Mar 2, 2017 •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading

ixmatus commented Mar 8, 2017 via email •

edited

Loading