unable to build images with gitlab runners and sysbox-runc as runtime

[samuelb]

Description

I'm having troubles running a buildah build ... from Gitlab-CI on a self-hosted runner with docker as executor and sysbox-runc as docker runtime. Following error is printed in the jobs output:

[...]
STEP 2/3: RUN apk add --no-cache python
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error detaching \"/var/tmp/buildah1882152697/mnt/rootfs\": no such file or directory"
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah1882152697/mnt/rootfs\": device or resource busy"
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah1882152697/mnt\": directory not empty"
error running subprocess: bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah1882152697/mnt/rootfs/dev: file exists
Error: deleting build container "96c3a9aad937e16e174792c1ec4edbae3907f1dd469ec0e02339c7eb1d029891": replacing mount point "/var/lib/containers/storage/overlay/76ca95dda33c9e8c343ddf776b4f99b8c9d2fb17c310c59f5cca0f285d9ee100/merged": device or resource busy: building at STEP "RUN apk add --no-cache python": exit status 1

When not using sysbox but the default docker runtime, the docker image build works just fine.
I'm able to reproduce it on Debian server as well as my local Arch installation.
Running the gitlab-runner in privileged mode instead doesn't change anything.

Steps to reproduce the issue:

1. On a machine which shall be a gitlab runner, install docker, sysbox-ce and gitlab-runner
2. Register the gitlab-runner with your project in Gitlab
3. Configure the docker sysbox-runc runtime:

   # cat /etc/docker/daemon.json
   {
     "runtimes": {
       "sysbox-runc": {
         "path": "/usr/bin/sysbox-runc",
         "runtimeArgs": ["--no-kernel-check"]
       }
     }
   }
   

4. Configure the gitlab-runner to use docker as executor with sysbox-runc as runtime

   # cat /etc/gitlab-runner/config.toml
   concurrent = 1
   check_interval = 0
   connection_max_age = "15m0s"
   shutdown_timeout = 0
   
   [session_server]
     session_timeout = 1800
   
   [[runners]]
     name = "..."
     url = "..."
     id = 408
     token = "..."
     executor = "docker"
     [runners.custom_build_dir]
     [runners.cache]
       MaxUploadedArchiveSize = 0
       [runners.cache.s3]
       [runners.cache.gcs]
       [runners.cache.azure]
     [runners.docker]
       tls_verify = false
       image = "docker:stable"
       privileged = false
       disable_entrypoint_overwrite = false
       oom_kill_disable = false
       disable_cache = false
       volumes = ["/cache"]
       shm_size = 0
       network_mtu = 0
       runtime = "sysbox-runc"
       devices = ["/dev/fuse"]
   

5. Ensure the config is running

   systemctl start sysbox
   systemctl restart docker
   systemctl restart gitlab-runner
   

6. Example Gitlab CI config .gitlab-ci.yml

   stages:
     - build
   
   build-job:
     stage: build
     image: quay.io/containers/buildah:v1.36.0
     script:
       - buildah version
       - buildah info
       - buildah build --file Dockerfile --manifest testimage .
   

7. Example Dockerfile

   FROM alpine
   
   RUN apk add --no-cache python3
   RUN apk add --no-cache curl
   

Describe the results you received:

The build fails with the error message show in the description.

Describe the results you expected:

buildah builds the image just fine without error.

Output of buildah version (from within a gitlab job):

Version:         1.36.0
Go Version:      go1.22.3
Image Spec:      1.1.0
Runtime Spec:    1.2.0
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.31.0
Git Commit:      
Built:           Mon May 27 13:11:54 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of buildah info (from within a gitlab job):

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "40"
        },
        "MemFree": 12764295168,
        "MemTotal": 33671274496,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "runner-wqxrcczgx-project-3228-concurrent-0",
        "kernel": "6.1.0-21-cloud-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "410m s",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.imagestore=/usr/lib/containers/storage",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,fsync=0"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Supports shifting": "true",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

Output of cat /etc/*release:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Output of uname -a:

Linux hostname 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Qwarctick]

Hello,

What happens when run try to run all commands by hand in the container ? This is to exclude a problem with the CI or the system.