[Self-Hosted] Custom docker image build failure : 'failed to mount /tmp/containerd-mount<XXXXXXXX>: operation not permitted'

[ccll]

Bug description

My .gitpod.Dockerfile builds successfully with running docker build ... directly in workspace container, but failed to build after commited to git and started a new workspace based on that new commit.

Related logs:

...
.gitpod.Dockerfile:3
--------------------
   1 |     FROM gitpod/workspace-python:latest
   2 |     
   3 | >>> RUN pyenv install 3.10.2
   4 |     RUN pyenv global 3.10.2
   5 |     
--------------------
error: failed to solve: failed to compute cache key: failed to mount /tmp/containerd-mount2739100871: operation not permitted
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","command":"build","error":"exit status 1","level":"error","message":"build failed","serviceContext":{"service":"bob","version":""},"severity":"ERROR","time":"2022-04-08T01:05:21Z"}
exit
...

Steps to reproduce

My host environment:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

(I know that only Ubuntu is officially supported, but my host is a legacy one and are running many production services on it now, and since Gitpod can almost run without much problem on it, I think there might be a way to make Debian as a community supported distro for Gitpod.)

Installed kernel 5.10 from buster-backports.

$ uname -a
Linux homelab-1 5.10.0-0.bpo.12-amd64 #1 SMP Debian 5.10.103-1~bpo10+1 (2022-03-08) x86_64 GNU/Linux

Kubernetes is a single node k3s cluster.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

$ crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/var/lib/rancher/k3s/data/4012316506613ee8c3cffc1e5b5eca706270685d33585804b257e93ea98d1917/bin"
    ],
    "PluginConfDir": "/var/lib/rancher/k3s/agent/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "flannel",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"delegate\":{\"forceAddress\":true,\"hairpinMode\":true,\"isDefaultGateway\":true},\"type\":\"flannel\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            }
          ],
          "Source": "{\n  \"name\":\"cbr0\",\n  \"cniVersion\":\"0.3.1\",\n  \"plugins\":[\n    {\n      \"type\":\"flannel\",\n      \"delegate\":{\n        \"hairpinMode\":true,\n        \"forceAddress\":true,\n        \"isDefaultGateway\":true\n      }\n    },\n    {\n      \"type\":\"portmap\",\n      \"capabilities\":{\n        \"portMappings\":true\n      }\n    }\n  ]\n}\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": null,
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "/var/lib/rancher/k3s/data/4012316506613ee8c3cffc1e5b5eca706270685d33585804b257e93ea98d1917/bin",
      "confDir": "/var/lib/rancher/k3s/agent/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "10010",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "rancher/mirrored-pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "containerdRootDir": "/var/lib/rancher/k3s/agent/containerd",
    "containerdEndpoint": "/run/k3s/containerd/containerd.sock",
    "rootDir": "/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/k3s/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.17.5",
  "lastCNILoadStatus": "OK"

Gitpod installer config file:

apiVersion: v1
authProviders: []
blockNewUsers:
  enabled: false
  passlist: []
certificate:
  kind: secret
  name: https-certificates
containerRegistry:
  inCluster: true
database:
  inCluster: true
disableDefinitelyGp: false
domain: "<my.domain.com>"
kind: Full
metadata:
  region: local
objectStorage:
  inCluster: true
observability:
  logLevel: info
openVSX:
  url: https://open-vsx.org
repository: eu.gcr.io/gitpod-core-dev/build
workspace:
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
  runtime:
    containerdRuntimeDir: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
    containerdSocket: /run/k3s/containerd/containerd.sock
    fsShiftMethod: fuse

.gitpod.yml

image:
  file: .gitpod.Dockerfile

tasks:
  - init: echo hello

.gitpod.Dockerfile

FROM gitpod/workspace-python:latest

RUN pyenv install 3.10.2
RUN pyenv global 3.10.2

Workspace affected

No response

Expected behavior

The docker image should build successfully just like running docker build ... in workspace container.

Example repository

No response

Anything else?

No response

[shaoye]

same issue here, did you solve it?

[ccll]

@shaoye Nope, still waiting the canonical solution.

Right now I workaround this by manually docker build & docker push to a container registry and reference the pushed image there.

[Seenox]

Same issue here and the aforementioned workaround wouldn't work for our use cases.
Could someone from GitPod Team help here, please?

[shaoye]

we switched to Ubuntu 2004 and it worked

[mrzarquon]

@ccli & @Seenox what is your containerd version (ctr version)?

I'm tracking down info so we can get a resolution on this.

[ccll]

@mrzarquon Sorry for the late reply.
Here is my ctr version

Client:
  Version:  1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  Go version: go1.13.15

Server:
  Version:  1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  UUID: 8c3f8d28-fa9a-429b-8a5c-ee3d7954b79c

[utam0k]

@ccll Hi, thanks for your information. Can you open this repository in your gitpod?
https://github.com/utam0k/gitpod-playground/tree/eStargz
https://${your_gitpod}/#https://github.com/utam0k/gitpod-playground/tree/eStargz

[utam0k]

This issue is brought into by eStargz. Fortunately, we just removed it.
#9492

[Seenox]

@mrzarquon @utam0k sorry for the late reply (and bumping the issue).
Here is my ctr version:

Client:
  Version:  1.5.4
  Revision: 69a4a55aa42353c71aae40e5bb27a412d8e3149b
  Go version: go1.13.15

Server:
  Version:  v1.4.6
  Revision: d71fcd7d8303cbf684402823e425e9dd2e99285d
  UUID: e092e812-9a86-44ad-b6b0-dbec0f674629

And I'm not able to open the example repository.
Checked in v2022.4.2, installed using kots.

[utam0k]

@Seenox cc: @gitpod-io/engineering-self-hosted
Thanks for your confirmation. It's turned out that the version of contained is probably irrelevant. The main branch has already included fixed PR. So, please wait for releasing the latest self-hosted version. If you will get the same error with the latest main branch, please feel free to reopen this issue.

[ccll]

Confirmed the issue was fixed in release-2022.05.0.5.
❤️

[Seenox]

Also confirming the issue has been fixed. Thanks!