Rootless Podman failes with "OCI permission denied" attempting to write to /proc/self/oom_score_adj on Fedora 38

[spacez320]

Issue Description

I'm attempting to build a FROM scratch container image which just copies and tries to invoke some Golang binaries, using Podman to run them. Building works fine, running produces this:

$ podman run -it --log-level=debug 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
...
DEBU[0000] ExitCode msg: "crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: permission denied\n\nopen executable: operation not permitted: oci permission denied"
Error: crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

Adding the --privileged and --group-add keep-groups do not produce different results.

I'm using the latest Podman as supplied by Fedora 38's RPMs, although admittedly Fedora is on release 39 so I might not be using what's latest-latest.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a Containerfile on Fedora 38.
2. Build the container image with podman build.
3. Attempt to run a container from the image with podman run.

The Containerfile I'm using looks like this, which is somewhat contextual to the thing I'm building, but is pretty simple:

FROM scratch

COPY dist/cryptarch_linux_amd64_v1 /cryptarch

ENTRYPOINT ["/cryptarch"]
CMD ["-h"]

Describe the results you received

Full debug output is below:

$ podman run -it --log-level=debug 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run -it --log-level=debug 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using boltdb as database backend
DEBU[0000] Initializing boltdb state at /home/matthew/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/matthew/.local/share/containers/storage
DEBU[0000] Using run root /run/user/4258/containers
DEBU[0000] Using static dir /home/matthew/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/4258/libpod/tmp
DEBU[0000] Using volume path /home/matthew/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 25
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6 (policy: missing)
DEBU[0000] Looking up image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage
DEBU[0000] Trying "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" ...
DEBU[0000] parsed reference into "[overlay@/home/matthew/.local/share/containers/storage+/run/user/4258/containers]@9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] Found image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" as "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage
DEBU[0000] Found image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" as "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage ([overlay@/home/matthew/.local/share/containers/storage+/run/user/4258/containers]@9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6)
DEBU[0000] Looking up image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage
DEBU[0000] Trying "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" ...
DEBU[0000] parsed reference into "[overlay@/home/matthew/.local/share/containers/storage+/run/user/4258/containers]@9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] Found image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" as "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage
DEBU[0000] Found image "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" as "9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6" in local containers storage ([overlay@/home/matthew/.local/share/containers/storage+/run/user/4258/containers]@9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6)
DEBU[0000] Inspecting image 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
DEBU[0000] exporting opaque data as blob "sha256:9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] exporting opaque data as blob "sha256:9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] Inspecting image 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
DEBU[0000] Inspecting image 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
DEBU[0000] Inspecting image 9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 15 for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270
DEBU[0000] parsed reference into "[overlay@/home/matthew/.local/share/containers/storage+/run/user/4258/containers]@9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] exporting opaque data as blob "sha256:9a6002c4e0d68c7e421951d7cc700d2e878e98dfeac2b4b3be7e0ed6bfd0acf6"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270"
DEBU[0000] Container "e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270" has work directory "/home/matthew/.local/share/containers/storage/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata"
DEBU[0000] Container "e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270" has run directory "/run/user/4258/containers/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating!        PID=3412012
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/home/matthew/.local/share/containers/storage/overlay/l/ZLUDOKNYEUIDOHS3VA66CTDUHU,upperdir=/home/matthew/.local/share/containers/storage/overlay/65c8ef87697b14a01770041951257d330639cc30df448b039d962afbd7babbb5/diff,workdir=/home/matthew/.local/share/containers/storage/overlay/65c8ef87697b14a01770041951257d330639cc30df448b039d962afbd7babbb5/work,userxattr,context="system_u:object_r:container_file_t:s0:c34,c289"
DEBU[0000] Made network namespace at /run/user/4258/netns/netns-d60023ee-9758-c503-1069-88ed242e2d72 for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270
DEBU[0000] Mounted container "e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270" at "/home/matthew/.local/share/containers/storage/overlay/65c8ef87697b14a01770041951257d330639cc30df448b039d962afbd7babbb5/merged"
DEBU[0000] Created root filesystem for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270 at /home/matthew/.local/share/containers/storage/overlay/65c8ef87697b14a01770041951257d330639cc30df448b039d962afbd7babbb5/merged
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/4258/netns/netns-d60023ee-9758-c503-1069-88ed242e2d72 tap0
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroups for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270 to user.slice:libpod:e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/home/matthew/.local/share/containers/storage/overlay/65c8ef87697b14a01770041951257d330639cc30df448b039d962afbd7babbb5/merged"
DEBU[0000] Created OCI spec for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270 at /home/matthew/.local/share/containers/storage/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270 -u e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270 -r /usr/bin/crun -b /home/matthew/.local/share/containers/storage/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata -p /run/user/4258/containers/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata/pidfile -n gifted_shtern --exit-dir /run/user/4258/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /run/user/4258/containers/overlay-containers/e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/matthew/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/4258/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/4258/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/matthew/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270]"
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270
DEBU[0000] Tearing down network namespace at /run/user/4258/netns/netns-d60023ee-9758-c503-1069-88ed242e2d72 for container e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270
DEBU[0000] Unmounted container "e08b807b52f453e0b4ca524825ba6a1ba2d4822e1d14f5313300994ec3c70270"
DEBU[0000] ExitCode msg: "crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: permission denied\n\nopen executable: operation not permitted: oci permission denied"
Error: crun: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

open executable: Operation not permitted: OCI permission denied
DEBU[0000] Shutting down engines

Describe the results you expected

Podman should run the container successfully.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 92.54
    systemPercent: 2.69
    userPercent: 4.77
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  freeLocks: 2032
  hostname: icarus.eudaimonia.io
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4258
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 4258
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.9-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2218663936
  memTotal: 33415528448
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11
      commit: 11f8d3dc9fc4bb8a0adcff5ba8bd340f24612701
      rundir: /run/user/4258/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc38.x86_64
    version: |
      pasta 0^20231004.gf851084-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/4258/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 7390621696
  swapTotal: 8589930496
  uptime: 337h 18m 30.00s (Approximately 14.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/matthew/.config/containers/storage.conf
  containerStore:
    number: 16
    paused: 0
    running: 0
    stopped: 16
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/matthew/.local/share/containers/storage
  graphRootAllocated: 998483427328
  graphRootUsed: 49156349952
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 26
  runRoot: /run/user/4258/containers
  transientStore: false
  volumePath: /home/matthew/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.3
  Built: 1704291091
  BuiltTime: Wed Jan  3 09:11:31 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

SELinux is enforcing but not reporting any errors or violations.

Additional information

This looks very similar to #9377 and #21051 but the conclusion of those are a little unclear.

[spacez320]

I should note that other podman run commands that don't execute copied binaries seem fine. I'm currently supplementing FROM scratch with a Go base image instead--FROM golang:1.20--and running that works fine under the same environment. However, attempting to copy the binaries under that context and execute them directly still fails.

[Luap99]

open executable: Operation not permitted: OCI permission denied

That is the real error you can ignore the oom_score_adj one. It usually means your binary is not executable so make sure the binary has the executable bit set.

[spacez320]

Ok, will dig in more to verify the script is actually runnable and come back with what I find, thanks @Luap99 .

[spacez320]

@Luap99 You're right, I was doing a few things wrong with my executable, this isn't a Podman problem as far as I can tell.

For anyone coming here,

• I wasn't COPYing what I thought I was, just a bad file path.
• I was mis-reading the oom_score_adj error as some sort of cause, when it was not, as suggested above.
• I was able to suss this out by exploring the container, swapping out FROM scratch with FROM <something-more-OS-like> and executing with podman -ti --entrypoint='' <recent-build> /bin/bash.

Sorry for the finger pointing. 😆

@Luap99 Does it make sense to add something about this to https://github.com/containers/podman/blob/main/troubleshooting.md ? By that I mean avoiding the confusion with oom_score_adj and motivating people to really check their thing is actually executable.

I would be happy to contribute, if so.

[Luap99]

If I try this I only get Error: crun: open executable: Permission denied: OCI permission denied which is expected.

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied is normal to see as rootless user when you add --log-level debug but here it somehow it got mixed up in the io from crun, likely because both printed at the same time?

this is my output with debug:

...
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-41845ceb44821d8545f505b270b1df261147f5ef6134a5c747c60423c3d69ff8.scope 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 41845ceb44821d8545f505b270b1df261147f5ef6134a5c747c60423c3d69ff8 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-593f4fde-0409-5c21-1198-75a020687e26 for container 41845ceb44821d8545f505b270b1df261147f5ef6134a5c747c60423c3d69ff8 
DEBU[0000] Unmounted container "41845ceb44821d8545f505b270b1df261147f5ef6134a5c747c60423c3d69ff8" 
DEBU[0000] ExitCode msg: "crun: open executable: permission denied: oci permission denied" 
Error: crun: open executable: Permission denied: OCI permission denied
DEBU[0000] Shutting down engines  

[spacez320]

Yeah, you're right @Luap99 , basically I saw the oom_score_adj and had a kneejerk reaction that it was a Podman issue. I think we can close this if we don't want to add a troubleshooting note (if we want to potentially consider this a "gotcha" other people might experience, but it's also just pure human error on my part).

[Luap99]

I mean you are more than welcome to submit a PR with some docs to describe the problem better if you think it helps others, i.e. add something to https://github.com/containers/podman/blob/main/troubleshooting.md or whatever place you looked.
Note that per podman(1) we even exit with exit code 126 in this case so even scripts could catch this problem automatically

[spacez320]

Oh, good call, I should have read the manual and checked the error code. Personally, I think that should have been enough. Will close, thanks a lot for your guidance!