podman-3.2.1 in container: cannot clone: Operation not permitted Error: cannot re-exec process #10802
Comments
|
My first thought is Seccomp - the Docker container could be blocking some flags of clone that we require. I believe that @cevich PTAL - do you recall any of the debugging you did for the linked issue? |
|
Yes or --security-opt label=disabled We are about to release a large blog on this, probably the week after the 4th. |
|
not using selinux -- unable to apply the options on the docker build ... |
Yes, my steps were:
Basically...what Dan said. IIRC, you (@mheon) and I had an IRC chat in which you confirmed: We deliberately bypass lots of stuff for the version sub-command. Apparently we still need to clone though 😞 In this particular case (with docker), I'm afraid the answer might be building with |
|
hmmmm...though...I do seem to remember playing around with running buildah builds inside a docker container. I seem to remember being able to pull it off by jamming upstream podman's seccomp-profile into docker. Lemmie see if I can find that work... |
|
...ugh, I can't find it. Sorry 😞 |
|
Prior to podman 3.2.1, we were able to build centos based docker containers which installed and ran podman to create our "podman docker base images" -- i.e. this all worked until last week, most recently with podman 3.1.2. Am I mistaken that this is a podman 3.2.1 change/regression/defect? Unfortunately, the kubic repos we use don't seem to have podman 3.1 anymore -- is there a different repo from which we can install podman 3.1.2 for centos as a temporary solution? |
|
If you can retest with 3.1.x, it would be interesting. I don't know if I'd consider this a regression given that there should be a reasonable expectation that the clone syscall works? An strace to get the exact failing syscall would also be helpful. |
|
Do you know where I can get podman 3.1.x packages for centos? They seem to be gone from the kubic yum repo. I can upload strace output if you like, but it's fairly huge. This is pretty easy to reproduce-- just run FROM centos:8
USER root
RUN dnf -y install strace
RUN dnf -y module disable container-tools \
&& dnf -y install 'dnf-command(copr)' \
&& dnf -y copr enable rhcontainerbot/container-selinux \
&& curl -sSL -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo
RUN dnf -y install podman
RUN strace podman --storage-driver=vfs info --debug |
|
By removing the kubic repo setup and using centos default podman packages, you can see this is successful using podman 3.0.1. Dockerfile: |
|
I think something is more fundamentally broken @rhatdan No docker build is needed to see the problem with 3.2.1: But with 3.1.2 it was fine: appears to be a 3.2.x issue: |
|
Try If you don't have SELinux no need for label=disable. |
|
tried both with and without |
|
works with |
|
seems |
|
|
|
Why would you need to? |
|
podman/buildah is not a mature, drop-in replacement for docker for container builds in jenkins pipelines. We support a common Jenkins CICD pipeline used across numerous teams to build hundreds of projects, most of which are not interested in making the switch to podman. We will need to support docker as a container build tool for some time as we try to increase podman adoption. Until the 3.2.x versions, we've been able to maintain a single Jenkins agent with both docker and podman (as well as a long list of other build tools) -- if we can't support docker and podman side-by-side, it's going to be difficult to increase podman adoption. For example, we use docker for most container builds, but have adopted Pod Manager container signing (skopeo copy --sign-by, podman push --sign-by) for image signing and verification. If running podman in docker/kubernetes isn't a supported configuration, we've got some rethinking to do. Was this a known/intentional change in 3.2.x? |
|
Sure, but I am asking why docker build has a relation to Podman. Are you using docker build against a Podman backend? |
|
No, we're using podman in a docker (kubernetes pod) container "back end". Our CICD infrastructure is Jenkins hosted in IBM Cloud -- our Jenkins agents are Centos 8 containers running as IBM Cloud kubernetes pods. These Centos 8 containers have podman installed (and docker and other tools). Within these pods, we perform docker builds. So, we have a container running in kubernetes that does docker builds (docker-in-docker) that need to execute podman commands. As of podman 3.2.1, we found this no longer works (see above) for example of docker build that no longer works when run in a container. We can't run a docker build with a Dockerfile that installs and executes podman commands. Follow up testing shows it's not jenkins or kubernetes -- or docker build. There seems to be a new requirement for podman 3.2.1 running in containers to have privileges that weren't needed before -- privileges we can't seem to grant to a docker build. Simplest example, we can't |
|
Thinking about this more... for our use case, I think we only need to use podman during a docker build when we're building our build agent (i.e. bootstrapping it). When building our jenkins agent, we install podman and then run |
|
Running podman within a Docker build is going to be difficult, for a lot of use cases. What you are seeing is we moved some of the commands under the user namespace when the CAP_SYS_ADMIN capability is not allowed. |
|
Oops docker build does not have --cap-add, but podman build does. |
|
OK, sounds like this was a choice -- makes me sad because it runs counter to the main "What is Podman" value prop: https://podman.io/whatis.html#what-is-podman-simply-put-alias-dockerpodman I'll work on refactoring our Jenkins agent build to not use podman in the docker build context. |
|
Well No if you want to have podman inside of a container talking to podman service on the host, then that will work without User Namespace. This is what Docker In Docker does, I believe. I don't think you can run docker containers within a docker build? podman --remote will work if you leak the podman socket from the host into the container. |
That might well not work -- never tried... I wasn't trying to run containers with podman. Previously, using
OK to close as "works as designed" |
|
Thanks, sorry about breaking your workflow. You can always get rid of docker engine on the host and switch to Podman everywhere. :^) |
|
Podman's not an easy swap in Jenkins workflows I'm afraid -- no great love for docker, but podman isn't CLI compatible enough to "alias podman=docker" yet. Hopefully in time! |
|
@rhatdan It looks like we're hitting this using the freedesktop.org ci-templates for GitLab CI builds. Those builds execute in a runc (because that's what GitLab upstream maintains for the CI runner) environment and then go on to use podman/buildah/skopeo (because they're nice tools). Unfortunately this doesn't work anymore, even when the CI containers are run with the equivalent of Not whinging here or demanding a revert, but hopefully this helps you understand some of the uses people have for podman and why running Podman under containerd is unavoidable sometimes. Thanks for all the tooling. |
|
My understanding of this is that |
|
Did we ever verify what caused this? Because I would lean towards a change in the Seccomp profile of the outer container, as opposed to a change to Podman itself - I cannot imagine rootless Podman (beyond trivial commands - |
I don't think so. I seem to recall a time when |
Used them to verify, initialize podman config in Jenkins jnlp agent docker build -- had to move all that to run time when the container is running |
|
There were no direct changes to the way we call However, digging through the changelog, I did find a very suspicious commit: Podman, as of 3.2.x, will (when run as root in an environment without It seems reasonable that a very limited subset of commands work without @rhatdan I believe this was your code, PTAL. |
|
As to why Podman-in-Podman works but Podman-in-Docker does not, our default Seccomp policy allows |
|
I made changes to fix a bug to move most of these commands into the user namespace which is why they are failing now. |
|
This is a real bug, should I just keep using 3.1.x? Is there an easy way to install just this version in a docker container? |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Following was working prior to release of podman 3.2.1 (i.e. worked until last week under podman 3.1.2)
Trying to build a centos-8-based container with podman and skopeo installed in a docker-in-docker (Jenkins agent running in kubernetes) environment... The larger context is Jenkins build of Jenkins jnlp agent with podman, docker, compilers, build tools etc, we use in our CICD pipelines to build containers, but the problem is reproducible with a small dockerfile running locally in docker (see below).
Seems similar to #10692 where the fix was to use podman instead of docker -- that's not an option for us at this time -- podman isn't mature enough to support the wide variety of container builds we support in our CICD pipelines. We've tried, but it's currently far from a drop-in replacement from docker -- we're trying! :-)
Steps to reproduce the issue:
Describe the results you received:
Full output of the docker build is included below, but the error in question is:
Describe the results you expected:
expected typical
podman infooutputAdditional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
can be reproduced locally with docker build running in a centos-8 based docker container (docker-in-docker):
The text was updated successfully, but these errors were encountered: