Filesystem isolation layer

[abitrolly]

/kind feature

Description

podman + SELinux = Leaky Abstraction. In my story containers provide isolated environment to help users concentrate on getting their app logic right and not think about low level details and permissions required to keep their systems stable and secure.

That worked good with Docker + Ubuntu, except for the part that docker process is itself is run as root, and that made users like me feel uneasy. podman appearance solved exactly this problem for me.

What makes my think that podman on Fedora is a leaky abstraction is that after one year of trying to adopt podman into my workflow I know a lot of information that I don't need to know, and yet I am still not there.

Yesterday I was able to solve the problem I started with a year ago thanks to the knowledge that I acquired about SELinux labelling, :z and :Z prefixes, USER and uid/gid mappings (that are not directly related to this issue, but learning them was necessary to remove unfitting pieces of different puzzle from my head). The problem is that python function shutil.copystat copies extended attributes and because container uses the same filesystem as host, SELinux denies this operation when you, for example, copystat files from /bin to keep them executable or read-only as before. There was a mistake in my volume mount command, which resulted in copying files from /bin instead of from mounted /src/bin. But I could figure it out only a year later when I did the same mistake.

I thought that the problem is solved. I copy files for my build inside container only from /src/bin. But today I realized that problem is not solved, because the build system copies system installed libs to build subtree, and SELinux vs copystat problem popped up again. I don't see who can I fix this, and I don't think that going down this rabbit hole is right way anyway.

What I really want from podman is Filesystem isolation level where the filesystem in container is completely isolated from the host, and volumes are no different from any other isolated container dir. No filesystem operation from inside container should trigger SELinux or other additional filesystem or kernel drivers on the host, and hence no SELinux properties (or any other kernel drivers) should be visible in container. If volume on the host contains those labels, the modifications to these files and labels should be done as if those files are created and modified by any user level program, such as vim.

Steps to reproduce the issue:

1. Build any snap with [stage-packages]

podman run --rm -it -v /home/user/linux:/src:Z -w /src/snapcrafting/amend yakshaveinc/snapcraft:core18 snapcraft

Describe the results you received:

...
  File "/snap/snapcraft/current/usr/lib/python3.5/shutil.py", line 252, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/snapcraft/current/usr/lib/python3.5/shutil.py", line 219, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/snap/snapcraft/current/usr/lib/python3.5/shutil.py", line 159, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/src/snapcrafting/amend/parts/amend/ubuntu/download/libgssapi3-heimdal_7.5.0+dfsg-1_amd64.deb'
...

Describe the results you expected:

...
Cleaning later steps and re-staging amend ('stage' property changed)
Priming amend 
'confinement' property not specified: defaulting to 'strict'
'grade' property not specified: defaulting to 'stable'
Snapping 'amend' |                                                                                                                                         
Snapped amend_0.1.0_amd64.snap

Additional information you deem important (e.g. issue happens only occasionally):

To keep the user story short - as a user of container I don't want to think about SELinux attributes on my host if my unprivileged container with a volume deep inside my home tries to play with some files.

The solution I tested for LXD on Ubuntu is to mount filesystem as 9p over the network through FUSE yakshaveinc/linux#32 I don't have money to keep focus and make a proper solution out of it (add encryption and integrate with LXD), but as proof of concept it works.

Output of podman version:

Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

[rhatdan]

Not really sure what is going on. I attempted to run your container on Fedora 31, but I am not sure what goes into /src/snapcrafting/amend

In side of the container SELinux should be seen as disabled so that the tools running in the container should not be trying to do SELinux operations. I think the shutil.py package is grabbing all XAttrs that is sees and attempting to apply them. And the SELinux label of the container is not allowed.

shutil.py should become more aware of what it is doing or you need to disable SELinux separation.

podman run -ti --security-opt label=disable

[rhatdan]

It looks like shuti.copyxattr now has support for ignoring these errors as least on Fedora 31?

/usr/lib64/python3.7/shutil.py

        """Copy extended filesystem attributes from `src` to `dst`.

        Overwrite existing attributes.

        If `follow_symlinks` is false, symlinks won't be followed.

        """

        try:
            names = os.listxattr(src, follow_symlinks=follow_symlinks)
        except OSError as e:
            if e.errno not in (errno.ENOTSUP, errno.ENODATA, errno.EINVAL):
                raise
            return
        for name in names:
            try:
                value = os.getxattr(src, name, follow_symlinks=follow_symlinks)
                os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
            except OSError as e:
                if e.errno not in (errno.EPERM, errno.ENOTSUP, errno.ENODATA,
                                   errno.EINVAL):
                    raise

[abitrolly]

I think the shutil.py package is grabbing all XAttrs that is sees and attempting to apply them. And the SELinux label of the container is not allowed.

I also think so. Why shutil.py is able to read those host level XAttrs in the first place? Placing requirements on what software should and should not call is what I call leaky abstraction of containerization for users.

Inside container the Python used is /snap/snapcraft/current/usr/lib/python3.5 and that's not going to change soon, because it is Python of LTS release.

--security-opt label=disable

Help says Turn off label separation for the container, but what does it exactly do?

[rhatdan]

It basically turns off SELinux labels, and runs container with an unconfined label.
Does the shutil.py inside of the container have the same code that I attached?

[abitrolly]

No. It was modified 7 months ago and it is not the same in Python 3.5 branch.

https://github.com/python/cpython/blame/master/Lib/shutil.py

https://github.com/python/cpython/blob/3.5/Lib/shutil.py

[rhatdan]

Well there is not much we can do other then have you disable SELinux separation until this is fixed. If this even works with SELinux separation turned off.

[abitrolly]

This is the container https://hub.docker.com/layers/yakshaveinc/snapcraft/core18/images/sha256-a7f7a8395ff9e060fe073652c8f3663c03229c6a0b8c01f89282081ab83fc2e1

[abitrolly]

@rhatdan does --security-opt label=disable mean that container will be able to read my SSH keys if I by mistake give it my home?

[rhatdan]

Yes. If it broke out to the file system, there would be nothing preventing the reading of these files if the process was running as your UID or as root.

[baude]

@rhatdan what gives on this one?

[rhatdan]

This needs to be fixed in the python3 package, I don't see anything for us to do here.

[abitrolly]

podman still needs filesystem isolation layer to be adopted by general public. I wouldn't bother writing this post long if it was no so critical. An ability to easily share work directory without reading a ton of info about UID/GID/SELinux/:x and :X prefixes/ACL like I did is critical for mass adoption. Otherwise from a public project it becomes a corporate policy.

[rhatdan]

@abitrolly Not sure what you mean but this, The same issue would happen in Docker with SELinux turned on. And it and Podman have been adopted by the "generate public".