Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rootless: readonly volume mount fails with unhelpful diagnostic #2312

Closed
edsantiago opened this issue Feb 11, 2019 · 15 comments
Closed

rootless: readonly volume mount fails with unhelpful diagnostic #2312

edsantiago opened this issue Feb 11, 2019 · 15 comments
Assignees
@giuseppe
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@edsantiago
Copy link
Member 

$ podman run --rm -v /tmp:/tmp:ro alpine date
container create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp\\\" to rootfs \\\"/home/esm/.local/share/containers/storage/vfs/dir/a1450cd2ae39b3a8f6e8263064ca2fada1406395912bd4bdc8bfd4c171de6858\\\" at \\\"/home/esm/.local/share/containers/storage/vfs/dir/a1450cd2ae39b3a8f6e8263064ca2fada1406395912bd4bdc8bfd4c171de6858/tmp\\\" caused \\\"operation not permitted\\\"\""
: internal libpod error

If run with -it, the error message is even less helpful:

$ podman run -it --rm -v /tmp:/tmp:ro alpine date
error reading container (probably exited) json message: EOF

There is no AVC in either case. Running with --log-level=debug produces no useful output (i.e. nothing different from a successful run). Removing :ro yields success.

podman-1.0.0-1.git82e8011.fc29 and runc-1.0.0-67.dev.git12f6a99.fc29 ; kernel 4.19.8-300.fc29.x86_64

Unfortunately I can't reproduce on a fresh virt.
@mheon
Copy link
Member

mheon commented Feb 11, 2019

I think we may have fixed this already on master

@edsantiago
Copy link
Member Author

I just built from 112a5ab and am still seeing the same problem.

edsantiago added a commit to edsantiago/libpod that referenced this issue Feb 13, 2019
@edsantiago
get_ci_vm : allow running without sudo
17a5f39 
More complicated than one would think. The first problem is that,
on certain (but not all) Fedora systems, podman cannot mount
volumes read-only (issue containers#2312). This is baffling, and since
it's not easily reproducible it's likely that the dev team
will not spend much effort on it. Workaround: instead of bind-
mounting /tmp read-only, bind-mount a *tempdir* (subdirectory)
read-write. This is actually cleaner in some ways but it
leads to complications with the paths we use and with cleanup.

Next, allow overriding the default image and allow asking
for no sudo:

    export GCLOUD_IMAGE=quay.io/edsantiago/gcloud_centos:latest
    export GCLOUD_SUDO=

(yes, that's an equal-sign and EOL. Just an empty string).

The third part, unfortunately, requires a custom image because
the as_dollar_user.sh script (the one that runs gcloud in a
container) is hardwired in a cevich image and needs tweaks
in order to detect rootless and avoid sudo.

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago edsantiago mentioned this issue Feb 13, 2019
Merged
@giuseppe
Copy link
Member

I am taking a look

@giuseppe
Copy link
Member

giuseppe commented Feb 13, 2019 •
edited
Loading

it looks like a bug in runc.

To verify it, I've tried another runtime:

$ podman --runtime /usr/local/bin/crun run -ti --rm -v /tmp:/tmp:ro alpine sh -c 'date; echo hello > /tmp/foo'
Wed Feb 13 16:19:20 UTC 2019
sh: can't create /tmp/foo: Is a directory

@edsantiago
Copy link
Member Author

I'm really confused now because I have two systems, one that reproduces the failure and one that works cleanly, and both have runc-1.0.0-67.dev.git12f6a99.fc29.x86_64

@giuseppe
Copy link
Member

is there any difference between the two systems? Same kernel?

@edsantiago
Copy link
Member Author

Lots of differences in installed packages; I've been trying to reduce that, but no luck.

Kernels that fail: 4.19.8-300.fc29, 4.20.3-200
Kernels that pass: 4.18.16-300, 4.20.6-200

@giuseppe
Copy link
Member

it fails on 4.20.6-200.fc29.x86_64 and runc-1.0.0-67.dev.git12f6a99.fc29.x86_64 as well

@edsantiago
Copy link
Member Author

it fails on 4.20.6-200.fc29.x86_64

Not for me. That seems to eliminate kernel as the source of the problem. (I just tested on 4.20.7-200.fc29; pass)

@edsantiago
Copy link
Member Author

@giuseppe what's your ~/.config/containers/storage.conf ? Mine is:

[storage]
  driver = "overlay"
  runroot = "/run/user/14904"
  graphroot = "/home/esm/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

I've also tried driver = "vfs" with mount_program = "". No difference. (I've also tried on LVM $HOME vs non-LVM; no difference)

@giuseppe
Copy link
Member

@edsantiago it is exactly the same, just different UID and home directory.

@rhatdan
Copy link
Member

rhatdan commented Feb 23, 2019

@edsantiago @giuseppe Is this resolved?

@edsantiago
Copy link
Member Author

I'm still seeing it on master - 0969d72

@giuseppe
Copy link
Member

I think it depends from runc, but I'll need to investigate it further

@giuseppe giuseppe mentioned this issue Feb 25, 2019
Merged
@giuseppe
Copy link
Member

PR here: #2432

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 25, 2019
@giuseppe
podman: fix ro bind mounts if no* opts are on the source
0f5ae3c 
This is a workaround for the runc issue:

opencontainers/runc#1247

If the source of a bind mount has any of nosuid, noexec or nodev, be
sure to propagate them to the bind mount so that when runc tries to
remount using MS_RDONLY, these options are also used.

Closes: containers#2312

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@kolyshkin kolyshkin mentioned this issue Mar 31, 2020
Merged
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@giuseppe giuseppe

Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@giuseppe @edsantiago @rhatdan @mheon