Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Subtle errors when running from inside a chroot #59

Closed
ihucos opened this issue Mar 31, 2019 · 10 comments
Closed

Subtle errors when running from inside a chroot #59

ihucos opened this issue Mar 31, 2019 · 10 comments

Comments

@ihucos
Copy link

ihucos commented Mar 31, 2019

When trying to run fuse-overlayfs from inside a chroot, there are hard to pin down, subtle errors.
This is some output of installing firefox inside arch linux with pacman.

(10/15) Creating temporary files...
Setting default ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Setting access ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
Cannot set file attribute for '/var/log/journal/remote', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
error: command failed to execute correctly
(11/15) Reloading device manager configuration...
  Skipped: Device manager is not running.
(12/15) Arming ConditionNeedsUpdate...
(13/15) Updating the info directory file...
(14/15) Updating the desktop file MIME type cache...
(15/15) Updating the MIME type database...

When using apt, there are errors about failed GPG checks, that I can not reproduce, it's kind of difficult to pin down.

This is interesting for nested containers. With unionfs-fuse I also get apparently the same kind of error: rpodgorny/unionfs-fuse#84 . But much more often.

Let me know if I should try to find an reproducible example or give me some direction to investigate.

@giuseppe
Copy link
Member

giuseppe commented Apr 1, 2019

are you running fuse-overlayfs as root? How did you configure the chroot?

@ihucos
Copy link
Author

ihucos commented Apr 1, 2019

Hey giuseppe,

apparently I am mistaken, this has nothing to do with running it inside a chroot, but seems to be due unsupported operations:

This are the straceed calls that caused the errors:

setxattr("/proc/self/fd/5", "system.posix_acl_access", "\2\0\0\0\1\0\7\0\377\377\377\377\4\0\5\0\377\377\377\377\10\0\5\0\346\3\0\0\10\0\5", 52, 0) = -1 EINVAL (Invalid argument)
openat(5, "log", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
ioctl(4, FS_IOC_GETFLAGS, 0x7ffdfd0c0668) = -1 ENOSYS (Function not implemented)
[pid  5100] openat(4, "tmp", O_RDONLY|O_NOCTTY|O_CLOEXEC|O_DIRECTORY) = 5
[pid  5100] fstat(5, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=110592, ...}) = 0
[pid  5100] fchownat(5, "", 0, 0, AT_EMPTY_PATH) = -1 EPERM (Operation not permitted)

That last one only happens sometimes, pacman than complains with fchownat() of /tmp failed: Operation not permitted - one time it was / instead of /tmp. I could not reproduce this by replaying the system calls.

Here is how fuse-overlayfs (as root and user), unionfs-fuse and pure overlay (unprivileged usage on ubuntu) compares.

$ cd /tmp
$ git clone git@github.com:ihucos/plash.git
Cloning into 'plash'...
remote: Enumerating objects: 146, done.
remote: Counting objects: 100% (146/146), done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 11134 (delta 85), reused 99 (delta 58), pack-reused 10988
Receiving objects: 100% (11134/11134), 16.16 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (6591/6591), done.
$ cd /tmp/plash/usr/local/bin/
$ export PLASH_DATA=/tmp/plashdata
$ plash init
$
$       # FUSE-OVERLAYFS #
$
$ echo "fuse-overlayfs" > /tmp/plashdata/config/union_taste
$ ./plash build -f archlinux --pacman apache
plash: fetching 100%
plash: extracting...
+ pacman -Sy --noconfirm apache
[...]
(2/3) Creating temporary files...
Setting default ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Setting access ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
Cannot set file attribute for '/var/log/journal/remote', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
error: command failed to execute correctly
(3/3) Arming ConditionNeedsUpdate...
--:
2
$
$       # UNIONFS-FUSE #
$
$ echo "unionfs-fuse" > /tmp/plashdata/config/union_taste
$ ./plash build -f archlinux --pacman apache --invalidate-layer
+ pacman -Sy --noconfirm apache
[...]
:: Running post-transaction hooks...
(1/3) Reloading system manager configuration...
  Skipped: Current root is not booted.
(2/3) Creating temporary files...
Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Invalid argument
Cannot set file attribute for '/var/log/journal/remote', value=0x00800000, mask=0x00800000, ignoring: Invalid argument
(3/3) Arming ConditionNeedsUpdate...
+ : invalidate cache with c8a56b6d-2bdc-4973-8268-034142d061a9
--:
3
$
$       # OVERLAY #
$
$ echo "overlay" > /tmp/plashdata/config/union_taste # on ubuntu ...
$ ./plash build -f archlinux --pacman apache --invalidate-layer
[...]
(1/3) Reloading system manager configuration...
  Skipped: Current root is not booted.
(2/3) Creating temporary files...
(3/3) Arming ConditionNeedsUpdate...
+ : invalidate cache with 1d6edb11-6955-4aaa-9102-07ebac0f62aa
--:
4
$
$       # PRIVILEGED FUSE-OVERLAYFS #
$
$ sudo PLASH_DATA=/tmp/plashdata2 ./plash init
$ sudo sh -c 'echo fuse-overlayfs > /tmp/plashdata2/config/union_taste'
$ sudo PLASH_DATA=/tmp/plashdata2 ./plash build -f archlinux --pacman apache
[...]
(1/3) Reloading system manager configuration...
  Skipped: Current root is not booted.
(2/3) Creating temporary files...
(3/3) Arming ConditionNeedsUpdate...
--:
2

So in conclusion when running as user, some operations seem not to be supported. So I am actually happy this bug is not what I initially thought and apparently more of a question of which system calls are supported and which are not :-)

Cheers,
Irae

@giuseppe
Copy link
Member

giuseppe commented Apr 1, 2019

@ihucos are you using a user namespace for running fuse-overlayfs? That is required for gaining the capabilities for writing these xattrs.

Also, another thing to try is not use tmpfs as the underlying storage. That could be the problem you are seeing

@ihucos
Copy link
Author

ihucos commented Apr 1, 2019

@ihucos are you using a user namespace for running fuse-overlayfs? That is required for gaining the capabilities for writing these xattrs.

Yes, I actually am.

$ plash -f archlinux -- id
uid=0(root) gid=0(root) groups=0(root),65534(nobody)
$ plash -f archlinux -- cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

Also, another thing to try is not use tmpfs as the underlying storage. That could be the problem you are seeing.

I did not know that. But actually the underlying storage is in the home directory. Hmmm...
This are the steps for setup:

  • Start new user and mount namespace
  • Overlay mount, with the layers, the changesdir and the mountpoint in the home directory
  • Recursive bind mounting /tmp, /home, /root, /etc/resolv.conf, /sys, /dev and /proc to the new root filesystem
  • chroot

@giuseppe
Copy link
Member

giuseppe commented Apr 2, 2019

currently ioctl are not supported. We could start adding them, for example:

diff --git a/main.c b/main.c
index 717b296..711bd7b 100644
--- a/main.c
+++ b/main.c
@@ -3713,6 +3713,29 @@ ovl_fsync (fuse_req_t req, fuse_ino_t ino, int datasync, struct fuse_file_info *
   fuse_reply_err (req, ret == 0 ? 0 : errno);
 }
 
+static void
+ovl_ioctl (fuse_req_t req, fuse_ino_t ino, int cmd, void *arg,
+           struct fuse_file_info *fi, unsigned flags,
+           const void *in_buf, size_t in_bufsz, size_t out_bufsz)
+{
+  int ret, res;
+  switch (cmd)
+    {
+    case FS_IOC_GETFLAGS:
+    case FS_IOC_SETFLAGS:
+      ret = ioctl (fi->fh, cmd, &res);
+      if (ret < 0)
+        fuse_reply_err (req, errno);
+      else
+        fuse_reply_ioctl (req, res, NULL, 0);
+      break;
+
+    default:
+        fuse_reply_err (req, ENOSYS);
+        break;
+    }
+}
+
 static struct fuse_lowlevel_ops ovl_oper =
   {
    .statfs = ovl_statfs,
@@ -3745,6 +3768,7 @@ static struct fuse_lowlevel_ops ovl_oper =
    .link = ovl_link,
    .fsync = ovl_fsync,
    .flock = ovl_flock,
+   .ioctl = ovl_ioctl,
   };
 
 static int

although FS_IOC_SETFLAGS doesn't seem to work for an unprivileged user. I've tried with a simple C program and tried to run it from a user namespace (not using fuse-overlayfs), I always get EPERM, at least on Fedora.

I think we should not compare to overlay in the kernel, as it runs with different privileges.

Could you try that patch if it makes any difference on your kernel?

@ihucos
Copy link
Author

ihucos commented Apr 2, 2019

That makes it different, I can't judge if better or worse:

(1/3) Reloading system manager configuration...
  Skipped: Current root is not booted.
(2/3) Creating temporary files...
Setting default ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Setting access ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Bad file descriptor
Cannot set file attribute for '/var/log/journal/remote', value=0x00800000, mask=0x00800000, ignoring: Bad file descriptor
error: command failed to execute correctly
(3/3) Arming ConditionNeedsUpdate...

At least for my use case it's generally working, which means that pacman does install the package. I don't know if there is some kind of "standard" of which filesystem operations are expected to work and for example package managers can live with without failing. Maybe it makes sense to document what operations are not supported by unprivileged fuse-overlayfs. The best thing is of course to just fix these things. At least as previously posted, with unionfs-fuse the setxattr call does not fail. So this could be possible somehow, I can't speak for the ioctl. If you want, I can try that simple script you wrote.

Here is just in case how to reproduce

$ git clone git@github.com:ihucos/plash.git
$ export PATH=plash/usr/local/bin:/where/is/fuse-overlayfs:$PATH
$ plash init
$ echo "fuse-overlayfs" | plash data tee config/union_taste
$ plash rm --from archlinux --pacman apache  # build container, then delete it immediately

@giuseppe
Copy link
Member

giuseppe commented Apr 2, 2019

are you able to do something like: unshare -r sh -c 'touch a && chattr +i a'? Does it succeed?

@ihucos
Copy link
Author

ihucos commented Apr 2, 2019

Unfortanly not

ihucos@macbook:~$ unshare -r sh -c 'touch a && chattr +i a'
chattr: Operation not permitted while setting flags on a

Some System information

ihucos@macbook:~$ stat / --format="%i"
2
ihucos@macbook:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
ihucos@macbook:~$ uname -a
Linux macbook 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

EDIT:
Different error message when trying on a tmpfs

ihucos@macbook:/tmp$ unshare -mr
root@macbook:~# mount -t tmpfs tmpfs /tmp
root@macbook:~# touch /tmp/a && chattr +i /tmp/a
chattr: Inappropriate ioctl for device while reading flags on /tmp/a

@giuseppe
Copy link
Member

giuseppe commented Apr 3, 2019

thanks for trying it out.

So the unprivileged user cannot use chattr, even when in a user namespace.

If the unprivileged user now gained that capability when using overlay from the kernel, then it looks like a security issue.

AFAICS, unionfs-fuse just ignore errors on the ioctl, could you confirm that if you try chattr +i on a unionfs-fuse mount, the underlying file doesn't really get that attribute?

So I think the most correct behaviour is the one showed by fuse-overlayfs with the patch I've proposed, if an user is not able to change the inode attrs then it must be reported as an error.

We could probably circumvent the issue by writing down the value in a xattr when the ioctl fails, it won't be portable but at least will work on the same fuse-overlayfs file system.

giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Apr 3, 2019
Closes: containers#59

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe
Copy link
Member

giuseppe commented Apr 3, 2019

I've cleaned up the patch and opened #61

There is no xattr emulation done, but it enables reading inode attributes, and setting them when running as root (on the host)

giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Apr 3, 2019
Closes: containers#59

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants