podman build with secret meet permission error, while podman run with secrets is ok.

[larrycai]

podman build with secret meet permission error

I follow the example in https://www.redhat.com/sysadmin/sensitive-data-containers to build the container image using secrets and meet permission error

Steps to reproduce the issue:

1. Create Dockerfile and touch one secret.txt

$ cat Dockerfile
FROM alpine
RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
RUN cat /mysecret

2. Run build command

$ podman build --secret id=mysecret,src=./secret.txt .
STEP 1/3: FROM alpine
STEP 2/3: RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
error running container: error from /usr/bin/runc creating container for [/bin/sh -c cat /mysecret]: time="2022-09-11T11:05:06+02:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"/var/tmp/buildah1779424060/mnt/buildah-bind-target-0\" to rootfs at \"/mysecret\" caused: operation not permitted"
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF 
Error: error building at STEP "RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret": error while running runtime: exit status 1

Describe the results you received:

see above, it reports permission error

Noticed some extra info about my storage

$ mount 
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/mapper/rootvg-root on / type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/vglocal-lvlocal on /local type xfs (rw,nosuid,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota,grpquota)

will nosuid be the issue? but it works for podman secret/run command

$ printf "top secrets" | podman secret create mysecret -
$ podman run --secret source=mysecret,type=env alpine printenv
..
mysecret=top secrets
$ podman run --secret source=mysecret,type=mount alpine mount
overlay on / type overlay (rw,relatime,lowerdir=/local/share/containers-user-1003/storage/overlay/l/2XJ3KVS3EGPTOXANZRHOQ4Z2NK,upperdir=/local/share/containers-user-
..
/dev/mapper/vglocal-lvlocal on /run/secrets/mysecret type xfs (rw,nosuid,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota,grpquota)
$ podman run --secret source=mysecret,type=mount alpine cat //run/secrets/mysecret
top secrets

Describe the results you expected:

In other machine, the Step 2 shall report ok

STEP 2/3: RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
top secrets
--> 10e64e9cba1
STEP 3/3: RUN cat /mysecret
cat: can't open '/mysecret': No such file or directory

Output of rpm -q buildah or apt list buildah:

podman-4.1.1-2.module+el8.6.0+15917+093ca6f8.x86_64

Output of podman version if reporting a podman build issue:

$ podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.17.7
Built:        Mon Jul 11 16:56:53 2022
OS/Arch:      linux/amd64
$ podman info
host:
  arch: amd64
  buildahVersion: 1.26.2
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-1.module+el8.6.0+14877+f643d2d6.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 87b7a9037cbd1d81288bdf2d6705dfda889f7cf9'
  cpuUtilization:
    idlePercent: 99.53
    systemPercent: 0.2
    userPercent: 0.27
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  hostname: xxx
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 64000
      size: 1
    - container_id: 1
      host_id: 3000655360
      size: 65535
    uidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 3000655360
      size: 65535
  kernel: 4.18.0-372.13.1.el8_6.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 12041859072
  memTotal: 33506668544
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.17.7
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1003/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 381h 55m 9.41s (Approximately 15.88 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
...
store:
  configFile: /home/bob/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /local/share/containers-user-1003/storage
  graphRootAllocated: 107317563392
  graphRootUsed: 8479199232
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1003/containers
  volumePath: /local/share/containers-user-1003/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1657551413
  BuiltTime: Mon Jul 11 16:56:53 2022
  GitCommit: ""
  GoVersion: go1.17.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

Output of cat /etc/*release:

NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
Red Hat Enterprise Linux release 8.6 (Ootpa)
Red Hat Enterprise Linux release 8.6 (Ootpa)

Output of uname -a:

Linux xxxx 4.18.0-372.13.1.el8_6.x86_64 #1 SMP Mon Jun 6 15:05:22 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[storage]
# Default Storage Driver, Must be set for proper operation.
driver = "overlay"
# Temporary storage location
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"
rootless_storage_path = "/local/share/containers-user-$UID/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

[storage.options.thinpool]

[larrycai]

Questions (if it is not a bug)

• is nosuid a issue for podman storage for local
• if yes, why it works for podman run command with secret

[larrycai]

I verified it using tmpfs for this (this VM is podman 3.4.2)

Start with nosuid

$ sudo mkdir /export 
$ sudo mount -t tmpfs -o size=1g xfs /export # default, there is no nosuid 
$ mount
xfs on /export type tmpfs (rw,relatime,size=1048576k)
$ podman build --no-cache --secret id=mysecret,src=./secret.txt .
STEP 2/3: RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
top secrets
--> 52661889c7c
STEP 3/3: RUN cat /mysecret

Now, I enable it and verified again

$ sudo mount -t tmpfs -o remount,nosuid xfs /export  # enable nosuid
$ mount
xfs on /export type tmpfs (rw,nosuid,relatime,size=1048576k)
$ podman build --no-cache --secret id=mysecret,src=./secret.txt .
TEP 2/3: RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
error running container: error from /usr/bin/runc creating container for [/bin/sh -c cat /mysecret]: time="2022-09-12T07:37:07
+02:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init cause
d: rootfs_linux.go:75: mounting \"/var/tmp/buildah676403037/mnt/buildah-bind-target-6\" to rootfs at \"/mysecret\" caused: ope
ration not permitted"
: exit status 1
Error: error building at STEP "RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret": error while running runtim
e: exit status 1

so do we have any document for this, and it will be good to show correct error msg when using nosuid file system for this command, like

=> podman build with secret doesn't work on nosuid based file system (graphRoot=/local)

[larrycai]

@rhatdan do u have time to give me help to comment this?

[flouthoc]

Hmm I am curious cause I think i have nosuid enabled on my fs but its just not xfs and secret works fine for me.

[larrycai]

it works for podman run, it doesn't work for podman build, can you find one VM to check using type tmpfs ;-) which I reproduced.

BTW: my VM has sudo permission is podman 3.4.2

[rhatdan]

@nalind @giuseppe Ideas?

Putting storage onto a nosuid file system will cause breakage within containers. For example. Running su or sudo within a container would break.

[giuseppe]

could you temporarily drop the nosuid bit from the mount and try again the command that has failed?

[larrycai]

for the issue I reported, so far I don't have permission to change (RHEL8/Podman 4.1). But I use tmpfs to reproduced it in another RHEL8/Podman 3.4.x, see my comments above.

[giuseppe]

my suggestion was to make sure it is really about nosuid and not about differences there can be with tmpfs.

Are you using /export for the root storage?

[larrycai]

• VM1 , I tested in using /tmp (it is under / storage), it works. For /local (nosuid), it doesn't work. I can't remove nosuid now
• VM2 (I have sudo): for tmpfs it is not for the root storage, just mounted as /export, I tried with/without nosuid, looks it is related

[giuseppe]

could you try changing OCI runtime? Do you get any error if you try --runtime /usr/bin/crun after you've installed crun?

[nalind]

It looks like the runtime is trying to ensure that the bind mount to the secret is suid/nosuid, exactly as we specify in the runtime config. build will bind-mount a file directly, so that fails. Secrets used by podman run are copied into a location that podman controls, so the suid/nosuid status of the source material never comes into play.
This can be confirmed by comparing the inode numbers of the source file and of the secret as seen from inside the container.

[larrycai]

@giuseppe good, this works

$ podman build --runtime=$HOME/bin/crun --secret id=mysecret,src=./secret.txt .
STEP 1/3: FROM alpine
STEP 2/3: RUN --mount=type=secret,id=mysecret,target=/mysecret cat /mysecret
top secrets
--> 3b2164563da
STEP 3/3: RUN cat /mysecret
cat: can't open '/mysecret': No such file or directory
Error: error building at STEP "RUN cat /mysecret": error while running runtime: exit status 1

BTW: I copied crun from other RHEL8, so it is placed under my $HOME

$ crun --version
crun version 1.4.1
commit: 802613580a3f25a88105ce4b78126202fef51dfb
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

I copied latest release crun-1.6, it is the same.

[larrycai]

Thank you so much @nalind @giuseppe @rhatdan !

Since RHEL8/podman is delivered with runc as default, therefore it will hurt the users who have local filesystem with nosuid (quite common option nowadays). I am ok to close this ticket if you don't want to go further.

And I recommend to

• add it in some troubleshooting guideline
• can we add more output for this error? podman build --secret to check graphRoot's file system

[rhatdan]

crun is available to RHEL8 users, we just can not change the default during a RHEL8 run. RHEL9 defaults to crun.

Please open a PR to modify the troubleshooting guide.

[larrycai]

Thank, close now. if there is a PR, I will link it here.

[imphil]

I stumbled across the same issue and opened a PR to add the information covered here to the podman docs at containers/podman#18878. PTAL.