SElinux denial with pasta when using non-default network

[GerbenWelter]

Issue Description

Starting some time ago (atleast some weeks ago, I didn't notice immediately) some containers failed to start. On the surface it looked like a permission error:

Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.421050946 +0200 CEST m=+0.511412041 system refresh
Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.528224539 +0200 CEST m=+0.618585634 volume create f40557fdf67662f46a6b353817d30394ada524d0b98851ec00d405cffde2829b
Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.44877864 +0200 CEST m=+0.539139735 image pull 70dc580695b9bd2908c49f67e3100ccefd160a07164bdf475e76177e85383444 docker.io/jacobalberty/unifi:v8
Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.564250981 +0200 CEST m=+0.654612076 container create b151472fb75ada95304ef7c1af0c48335bea1905a35a8c8ce1eccd93436d171f (image=docker.io/jacobalberty/unifi:v8, name=unifi, org.opencontainers.image.revision=74c76a44020889cb2dbdcbe9a8485afe46e820f2, org.opencontainers.image.source=https://github.com/jacobalberty/unifi-docker, maintainer=Jacob Alberty <jacob.alberty@foundigital.com>, org.opencontainers.image.licenses=MIT, org.opencontainers.image.description=Unifi Docker files, org.opencontainers.image.created=2024-11-10T12:42:27.162Z, PODMAN_SYSTEMD_UNIT=unifi.service, org.opencontainers.image.url=https://github.com/jacobalberty/unifi-docker, org.opencontainers.image.ref.name=ubuntu, io.containers.autoupdate=registry, org.opencontainers.image.title=unifi-docker, org.opencontainers.image.version=v8.6.9)
Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.772399502 +0200 CEST m=+0.862760597 container remove b151472fb75ada95304ef7c1af0c48335bea1905a35a8c8ce1eccd93436d171f (image=docker.io/jacobalberty/unifi:v8, name=unifi, PODMAN_SYSTEMD_UNIT=unifi.service, io.containers.autoupdate=registry, org.opencontainers.image.title=unifi-docker, org.opencontainers.image.version=v8.6.9, org.opencontainers.image.created=2024-11-10T12:42:27.162Z, org.opencontainers.image.revision=74c76a44020889cb2dbdcbe9a8485afe46e820f2, org.opencontainers.image.source=https://github.com/jacobalberty/unifi-docker, org.opencontainers.image.url=https://github.com/jacobalberty/unifi-docker, maintainer=Jacob Alberty <jacob.alberty@foundigital.com>, org.opencontainers.image.description=Unifi Docker files, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.licenses=MIT)
Jun 19 19:05:04 nuc.welter.lan podman[1540]: 2025-06-19 19:05:04.804979726 +0200 CEST m=+0.895340820 volume remove f40557fdf67662f46a6b353817d30394ada524d0b98851ec00d405cffde2829b
Jun 19 19:05:04 nuc.welter.lan podman[1540]: Error: setting up Pasta: pasta failed with exit code 1:
Jun 19 19:05:04 nuc.welter.lan podman[1540]: Couldn't open PID file /tmp/containers-user-1003/containers/networks/rootless-netns/rootless-netns-conn.pid: Permission denied
Jun 19 19:05:04 nuc.welter.lan systemd[1093]: unifi.service: Main process exited, code=exited, status=126/n/a

After digging deeper I found that it was actually SElinux denying pasta to create/read the PID file:

type=AVC msg=audit(1750358869.815:377): avc:  denied  { search } for  pid=5416 comm="pasta" name="containers-user-1003" dev="tmpfs" ino=33 scontext=unconfined_u:unconfined_r:pasta_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:container_var_run_t:s0 tclass=dir permissive=0

Temporarily setting SElinux to permissive made the containers start again. So setting SElinux to enforcing again and creating and loading the following SElinux policy made the containers start all the time:

module mypasta 1.0;

require {
	type container_var_run_t;
	type pasta_t;
	class dir { add_name open read search watch write };
	class file { create open write };
}

#============= pasta_t ==============

allow pasta_t container_var_run_t:dir { add_name open read search watch write };
allow pasta_t container_var_run_t:file { create open write };

But needing this policy all of the sudden kept bugging me. Searching bugzillas, this project, etc I found no similar recent problems so it must be a local issue. Then I finally noticed the difference between the failing containers and non-failing container: The failing containers were not using the default network. For more isolation I had created a podman network per container. This had always worked until one of the weekly updates in the past few weeks (months?) made the containers fail to start but I didn't notice because they run on a headless server and automatically start on boot. Editing the systemd unit file to not use the '--network' parameter made the failing containers start again.

Steps to reproduce the issue

Steps to reproduce the issue

1. Install Fedora Silverblue
2. Create custom podman network (podman network create )
3. Start container with --network <name> parameter
4. Running an network isolated container

Describe the results you received

Container fails to start due to permission/SElinux errors.

Describe the results you expected

Running containers using isolated networks. The podman config hasn't been changed but started failed after applying host (Fedora Silverblue 42) updates somewhere in the past weeks.

podman info output

host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  - dmem
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 98.59
    systemPercent: 0.57
    userPercent: 0.84
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: silverblue
    version: "42"
  eventLogger: journald
  freeLocks: 2048
  hostname: nuc.welter.lan
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.14.11-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 13471715328
  memTotal: 16160579584
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/sbin/pasta
    package: passt-0^20250611.g0293c6f-1.fc42.x86_64
    version: |
      pasta 0^20250611.g0293c6f-1.fc42.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/sbin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 2h 17m 57.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 248355225600
  graphRootUsed: 146591784960
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.5.1
  BuildOrigin: Fedora Project
  Built: 1749081600
  BuiltTime: Thu Jun  5 02:00:00 2025
  GitCommit: 850db76dd78a0641eddb9ee19ee6f60d2c59bcfa
  GoVersion: go1.24.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Running baremetal on a Fedora Silverblue host.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[lsm5]

@sbrivio-rh PTAL

[sbrivio-rh]

Thanks @GerbenWelter for reporting this! This is the same as https://bugzilla.redhat.com/show_bug.cgi?id=2373054. As it only happens for containers that were started before a recent upgrade of passt-selinux, a workaround would be to restart them.

I would suggest to track this on the existing Fedora ticket, but we can also leave this one open if it's practical, and I'll take care of closing it once the issue is fixed.

[GerbenWelter]

@sbrivio-rh Ahh, I failed to find that one, but because I use Fedora Silverblue I need to reboot to use the updated packages. So the containers have definitely been restarted before. Naturally I also restarted the containers during troubleshooting. So rebooting/restarting doesn't fix this issue.

[sbrivio-rh]

Oops, sorry, according to https://bugzilla.redhat.com/show_bug.cgi?id=2373054#c3 the only way to fix this for existing containers is podman system reset. Well, you can also do, as root (untested, but I'm fairly sure it should work, minus typos):

chcon system_u:object_r:pasta_pid_t:s0 /tmp/storage-run-1003/containers/networks/rootless-netns/rootless-netns-conn.pid

[GerbenWelter]

The path you mentioned (and was also mentioned in the bugzilla) was not present on my system but a similar one was so I adapted your command to:

chcon system_u:object_r:pasta_pid_t:s0 /tmp/containers-user-1003/containers/networks/rootless-netns

(no pid file, directory was empty)

but that did not fix the problem. So I did podman system reset, recreated the extra podman network and the container started again. I did notice after the container was running again the directory /tmp/containers-user-1003/containers/networks was not present anymore. So I guess there was a mismatch between podman and the container definition and that's why the reset was needed?

So the fix is clear. If needed I can leave another container in the same state so I can gather additional information if needed. Just let me know.

[Luap99]

/tmp/containers-user-1003/containers

That is general a sign that the first time you used podman on that user you had no proper systemd login session (no $XDG_RUNTIME_DIR defined). Since the root/runnroot should never changed podman then caches that "incorrect" dir forever (well until podman system reset). Now with the proper XDG_RUNTIME_DIR runroot it starts working as that seems to have the proper labels.

What I don't get here or in https://bugzilla.redhat.com/show_bug.cgi?id=2373054 why is the rundir labelled container_var_run_t? I don't see a rule describing that and by default /tmp paths seems to have user_tmp_t which is what we would expect and where the transition rule would relabel the paths properly

[matteo-gambarutti]

I'm afraid I got hit by this issue as well (at least I think as a see a few entries in SELinux logs).
I'm wondering if I should rollback or run podman system reset.

1. How destructive is this command? I only use bind mounts so containers data should be safe but I rather check here before I proceed.
2. Do I have to stop all containers before performing the reset?
3. I use only run rootless containers, do I need to run this command as root or not?

[Luap99]

A simple reboot should be enough, if the selinux labels are still incorrect after a reboot something really strange is happening and we need to figure out why some users have different labels on the /tmp paths.
In any case what is the label on the runroot dir ls -Zd $(podman info --format {{.Store.RunRoot}})? Before and after the reboot, is there any difference?

podman system reset must be run as the same user and will delete all containers, images, volumes, ... So if you have important volumes (podman volume ls) then make sure they are backed up. However if you use bind mounts then these directories are not deleted.

[wdouglascampbell]

I don't know if this is safe but it seems to work for me instead of using podman system reset which was too destructive for my situation. I did the following and it seemed to correct the RunRoot location for me.

1. Go to https://github.com/br0xen/boltbrowser/releases
2. Download the latest Linux binary (e.g. boltbrowser.linux64)
3. Make it executable: chmod +x boltbrowser.linux64
4. Shutdown all containers (in my case, when I shut them down they are removed)
5. Backup current ~/.local/share/containers/storage/libpod/bolt_state.db file (just in case)
6. Edit bolt_state.db

./boltbrowser.linux64 ~/.local/share/containers/storage/libpod/bolt_state.db

7. Navigate to + runtime-config and press Enter to expand it.
8. Navigate to run-root and press Enter to edit its value.
9. Change from /tmp path to /run/user path.
10. Navigate to tmp-dir and press Enter to edit its value.
11. Change from /tmp/run-[uid]/libpod/tmp to /run/user/[uid]/libpod/tmp
12. Exit boltbrowser

In my case once I finished this and then started my containers, they didn't have the permissions errror from SELinux and their files were no longer being put under /tmp and instead were under /run/user.

Hope that is helpful to someone else who doesn't want to to podman system reset.