All builds fail with "runc run failed: unable to start container process"

[janvda]

Description

Since a new version of the docker.io/moby/buildkit image got pulled, I can no longer build any images.

I am always getting errors like:

=> ERROR [2/7] RUN set -ex && apk --no-cache add sudo openssh-client &&     echo "node-red ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoe  0.5s
------
 > [2/7] RUN set -ex && apk --no-cache add sudo openssh-client &&     echo "node-red ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers:
#0 0.185 runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument
------
failed to solve: process "/bin/sh -c set -ex && apk --no-cache add sudo openssh-client &&     echo \"node-red ALL=(ALL) NOPASSWD: ALL\" >> /etc/sudoers" did not complete successfully: exit code: 1

Note that I have

• upgraded colima
• rebooted machine
• removed buildkit image
• upgraded macos

but I am still facing the same problem.

Maybe it is related to the fact that my qemu-system-x86_64 is not signed.

mac-jan:my-question-generator jan$ codesign --verify /Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64
/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64: code object is not signed at all
In architecture: x86_64
mac-jan:my-question-generator jan$ 

Version

Colima Version: HEAD-afe81c4 git commit: afe81c4
Lima Version: 0.17.2
Qemu Version: 8.1.0

Operating System

• macOS Intel <= 12 (Monterrey)
• macOS Intel >= 13 (Ventura)
• macOS M1 <= 12 (Monterrey)
• macOS M1 >= 13 (Ventura)
• Linux

Output of colima status

INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/jan/.colima/default/docker.sock

Reproduction Steps

1. start colima (colima start)
2. assuring that my docker context is pointing to colima (unix:///Users/jan/.colima/default/docker.soc)
3. try to build the container using docker-compose (e.g. docker-compose -f docker-compose.yml up -d --build)

Expected behaviour

The container should be build without errors.

Additional context

When starting colima it is reporting also an error about qemu-system-x86_64 is not properly signed

mac-jan:my-question-generator jan$ colima start
INFO[0000] starting colima                              
INFO[0000] runtime: docker                              
INFO[0000] preparing network ...                         context=vm
INFO[0000] starting ...                                  context=vm
> Using the existing instance "colima"
> "QEMU binary \"/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64\" is not properly signed with the \"com.apple.security.hypervisor\" entitlement" error="failed to run [codesign --verify /Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64]: exit status 1 (out=\"/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64: code object is not signed at all\\nIn architecture: x86_64\\n\")"
> You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually. See https://github.com/lima-vm/lima/issues/1742 .
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/jan/.lima/colima/serial*.log")
> SSH Local Port: 50980
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"

No response

[janvda]

I have signed qemu-system-x86_64 through below command but this didn't fix the problem.
So that is not the root cause.

mac-jan:my-question-generator jan$ cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF
mac-jan:my-question-generator jan$ codesign --sign - --entitlements entitlements.xml --force /Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64

Wen trying to build image again after signin I am still getting error:

 > [2/3] RUN apt-get update -y &&     apt-get install -y git nano wget &&     pip install --upgrade pip:
#0 0.263 runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument

[janvda]

Setting rc_cgroup_mode to "unified" also didn't fix the issue (see kubernetes-sigs/kind#3277 (comment))

[janvda]

I tried to downgrade QEMU to v8.0.3 as proposed by (#786 (comment)) without success.

Here the steps followed to downgrade:

mac-jan:my-question-generator jan$ brew uninstall --ignore-dependencies qemu
Uninstalling /usr/local/Cellar/qemu/8.1.0_1... (162 files, 528.8MB)
mac-jan:my-question-generator jan$ curl -OSL https://raw.githubusercontent.com/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5461  100  5461    0     0  17456      0 --:--:-- --:--:-- --:--:-- 17846
mac-jan:my-question-generator jan$ brew install ./qemu.rb
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
...
==> Running `brew cleanup qemu`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /Users/jan/Library/Caches/Homebrew/qemu--8.0.3... (103.8MB)
==> Upgrading 1 dependent of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
qemu 8.0.3 -> 8.1.0_1
Error: Formula installation already attempted: qemu
mac-jan:my-question-generator jan$

... it seems that the downgrade didn't succeed.

[janvda]

kubernetes-sigs/kind#3277 seems to report same error.

[saghul]

FWIW I'm run into the same problem while trying to migrate away from Docker Desktop.

I'm on macOS 12 still, on an M1. Building the arm64 part works, but the amd64 part gives me the same error.

Running a container with docker run --platform linux/amd64 does work though.

[janvda]

As work around: instead of using colima i have used the docker template provided by lima.
More precisely I have executed following steps:

1. identify all lima instances : limactl list
2. stop all running lima instances: limactl stop <instance name>
3. delete all lima instances: limactl delete <instance name>
4. create and start the docker lima instance: limactl start template://docker
5. the output of previous command will also tell you how you can set the docker context.
6. Now I could properly build images without errors.

[saghul]

Thanks for the work around @janvda ! That worked great.

[onyxraven]

related: #764

[dramich]

Having run into this as well, I think the proper fix is in the comment in the issue mentioned above: #764 (comment) which is setting up a builder using a rootless image

[Skarlso]

FWIW, for kind, reverting to v0.19.0 fixed my issue with kind reporting this problem:

kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
➜  ~ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS                       NAMES
ffa300dd383b   kindest/node:v1.27.1   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   127.0.0.1:49982->6443/tcp   kind-control-plane

[abiosoft]

This is getting fixed in imminent v0.6.0.