CentOS 7 with SELinux prevents containers to run

[Fodoj]

Version:

k3s version v1.17.4+k3s1 (3eee8ac3)

[root@ip-10-0-1-116 centos]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@ip-10-0-1-116 centos]# uname -r
3.10.0-1062.12.1.el7.x86_64

 cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = true
  sandbox_image = "docker.io/rancher/pause:3.1"

[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/6a3098e6644f5f0dbfe14e5efa99bb8fdf60d63cae89fdffd71b7de11a1f1430/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

Installation script:

#!/bin/bash

set -e

yum install -y container-selinux selinux-policy-base iptables-services
rpm -i https://rpm.rancher.io/k3s-selinux-0.1.1-rc1.el7.noarch.rpm

curl -sfL https://get.k3s.io | sh -

Describe the bug

Take this YAML:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-2
  namespace: default
spec:
  containers:
  - image: postgres:9.6.9
    imagePullPolicy: IfNotPresent
    name: svc-0
    resources: {}
    securityContext:
      privileged: false
    stdin: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /builds
      name: repo
  volumes:
  - emptyDir: {}
    name: repo

Then kubectl create -f pod.yaml

Note it happens with multiple other images as well, not just postgresql. But postgresql is easiest to use to re-produce

Expected behavior

postgresql starts fine

Actual behavior

test-pod-2   0/1     CrashLoopBackOff   4          2m44s

In logs:

chown: changing ownership of '/var/lib/postgresql/data': Permission denied

In audit logs (parsed with sealert for better output, I installed setroubleshoot-server for this to work):

sealert -a /var/log/audit/audit.log > /var/log/sealert.log 

found 3 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /bin/chown from setattr access on the directory f3b27ddb70e2b0702f99417e9985ee6185a18b49162930635668eb923e85b346.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that chown should be allowed setattr access on the f3b27ddb70e2b0702f99417e9985ee6185a18b49162930635668eb923e85b346 directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'chown' --raw | audit2allow -M my-chown
# semodule -i my-chown.pp


Additional Information:
Source Context                system_u:system_r:container_t:s0:c297,c472
Target Context                system_u:object_r:container_var_lib_t:s0
Target Objects                f3b27ddb70e2b0702f99417e9985ee6185a18b491629306356
                              68eb923e85b346 [ dir ]
Source                        chown
Source Path                   /bin/chown
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           coreutils-8.22-24.el7.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-252.el7_7.6.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     ip-10-0-1-116.eu-central-1.compute.internal
Platform                      Linux ip-10-0-1-116.eu-central-1.compute.internal
                              3.10.0-1062.12.1.el7.x86_64 #1      SMP Tue Feb 4
                              23:02:59 UTC 2020 x86_64 x86_64
Alert Count                   1
First Seen                    2020-04-22 07:13:29 UTC
Last Seen                     2020-04-22 07:13:29 UTC
Local ID                      de688115-d748-40cf-928a-fcedfeb607ad

Raw Audit Messages
type=AVC msg=audit(1587539609.165:373): avc:  denied  { setattr } for  pid=23064 comm="chown" name="f3b27ddb70e2b0702f99417e9985ee6185a18b49162930635668eb923e85b346" dev="nvme0n1p1" ino=130063614 scontext=system_u:system_r:container_t:s0:c297,c472 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0


type=SYSCALL msg=audit(1587539609.165:373): arch=x86_64 syscall=fchownat success=no exit=EACCES a0=ffffff9c a1=559935030640 a2=3e7 a3=ffffffff items=0 ppid=23037 pid=23064 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=chown exe=/bin/chown subj=system_u:system_r:container_t:s0:c297,c472 key=(null)

Hash: chown,container_t,container_var_lib_t,dir,setattr

--------------------------------------------------------------------------------

SELinux is preventing /bin/busybox from create access on the file config.jsonIHImJc.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that busybox should be allowed create access on the config.jsonIHImJc file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'sed' --raw | audit2allow -M my-sed
# semodule -i my-sed.pp


Additional Information:
Source Context                system_u:system_r:container_t:s0:c297,c472
Target Context                system_u:object_r:container_var_lib_t:s0
Target Objects                config.jsonIHImJc [ file ]
Source                        sed
Source Path                   /bin/busybox
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-252.el7_7.6.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     ip-10-0-1-116.eu-central-1.compute.internal
Platform                      Linux ip-10-0-1-116.eu-central-1.compute.internal
                              3.10.0-1062.12.1.el7.x86_64 #1      SMP Tue Feb 4
                              23:02:59 UTC 2020 x86_64 x86_64
Alert Count                   5
First Seen                    2020-04-22 07:13:39 UTC
Last Seen                     2020-04-22 07:13:39 UTC
Local ID                      aeb29d00-9933-4033-80f6-48a1e159876d

Raw Audit Messages
type=AVC msg=audit(1587539619.958:380): avc:  denied  { create } for  pid=23170 comm="sed" name="config.jsonIHImJc" scontext=system_u:system_r:container_t:s0:c297,c472 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0


type=SYSCALL msg=audit(1587539619.958:380): arch=x86_64 syscall=open success=no exit=EACCES a0=55e2d9968880 a1=c2 a2=180 a3=0 items=0 ppid=23138 pid=23170 auid=4294967295 uid=2000 gid=2000 euid=2000 suid=2000 fsuid=2000 egid=2000 sgid=2000 fsgid=2000 tty=(none) ses=4294967295 comm=sed exe=/bin/busybox subj=system_u:system_r:container_t:s0:c297,c472 key=(null)

Hash: sed,container_t,container_var_lib_t,file,create

--------------------------------------------------------------------------------

SELinux is preventing /bin/chown from setattr access on the directory 6616f1c5fc8eb3012dad9afcf78bf1807fd8c80a5dee73c966fed4b2ea4989ed.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that chown should be allowed setattr access on the 6616f1c5fc8eb3012dad9afcf78bf1807fd8c80a5dee73c966fed4b2ea4989ed directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'chown' --raw | audit2allow -M my-chown
# semodule -i my-chown.pp


Additional Information:
Source Context                system_u:system_r:container_t:s0:c350,c710
Target Context                system_u:object_r:container_var_lib_t:s0
Target Objects                6616f1c5fc8eb3012dad9afcf78bf1807fd8c80a5dee73c966
                              fed4b2ea4989ed [ dir ]
Source                        chown
Source Path                   /bin/chown
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           coreutils-8.22-24.el7.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-252.el7_7.6.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     ip-10-0-1-116.eu-central-1.compute.internal
Platform                      Linux ip-10-0-1-116.eu-central-1.compute.internal
                              3.10.0-1062.12.1.el7.x86_64 rancher/k3s#1 SMP Tue Feb 4
                              23:02:59 UTC 2020 x86_64 x86_64
Alert Count                   6
First Seen                    2020-04-22 07:19:53 UTC
Last Seen                     2020-04-22 07:22:40 UTC
Local ID                      f3b2a883-6640-4c53-8fa1-6dee43480a0b

Raw Audit Messages
type=AVC msg=audit(1587540160.864:412): avc:  denied  { setattr } for  pid=24995 comm="chown" name="6616f1c5fc8eb3012dad9afcf78bf1807fd8c80a5dee73c966fed4b2ea4989ed" dev="nvme0n1p1" ino=193128242 scontext=system_u:system_r:container_t:s0:c350,c710 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0


type=SYSCALL msg=audit(1587540160.864:412): arch=x86_64 syscall=fchownat success=no exit=EACCES a0=ffffff9c a1=560265fd9640 a2=3e7 a3=ffffffff items=0 ppid=24981 pid=24995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=chown exe=/bin/chown subj=system_u:system_r:container_t:s0:c350,c710 key=(null)

Hash: chown,container_t,container_var_lib_t,dir,setattr

Additional context / logs

Putting SELinux to Permissive mode resolves the issue.

[erikwilson]

Thanks for reporting @Fodoj. I was able to verify this behavior is a problem by comparing it to running docker with selinux enabled.

[ShylajaDevadiga]

Issue is resolved. Verified using k3s v1.18.2-rc4+k3s1

[root@ip-172-31-11-166 ~]# k exec -it test-pod-2 sh
#  ls -lZ /var/lib/postgresql/data
total 56
drwx------. 5 postgres postgres system_u:object_r:container_file_t:s0:c332,c897    41 May  5 20:53 base
drwx------. 2 postgres postgres system_u:object_r:container_file_t:s0:c332,c897  4096 May  5 20:54 global

[Fodoj]

I tried with 1.18.2. The postgres issue from the example above seems to be fixed, by I encounter another problem.

I got it with my custom image - quay.io/repository/fodoj/mattermost - you can to re-produce it, it's a publicly available image (just a packaging around Mattermost). But I can also try to provide a more concrete test case, once I have more time to do so.

The issue I have is this:

There is a user, mattermost, part of the group - mattermost.

There is a file inside the image which is owned by mattermost:mattermost and has rw permissions for owner. Despite this permissions and the user inside the container being mattermost, I get permission denied error, which can be only SELinux related from the first glance.

[erikwilson]

Thanks for the info @Fodoj! If you are able to file a new issue using the mattermost image that would be greatly appreciated. Would also be nice to see a comparison of process & mount labels using k3s w/ docker & selinux enabled vs using k3s w/ embedded containerd.