OCI runtime create failed - open /proc/sys/kernel/domainname: read-only file system: unknown

[jankeromnes]

Bug description

A friend of mine is trying to use Gitpod for a customer project using openldap and docker-compose. The Docker image being used is osixia/openldap:stable.

They're currently hitting this error when running docker-compose up:

ERROR: for openldap  Cannot start service openldap: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: read-only file system: unknown

Is this a potential bug in Gitpod's Docker-in-Docker setup?

Steps to reproduce

1. Have a docker-compose.yml similar to this one:

docker-compose.yml

version: "3.9"
services:
  openldap:
    image: osixia/openldap:stable
    container_name: openldap
    environment:
      LDAP_LOG_LEVEL: "256"
      LDAP_ORGANISATION: "Example Inc."
      LDAP_DOMAIN: "exampledomain"
      LDAP_BASE_DN: ""
      LDAP_ADMIN_PASSWORD: "admin"
      LDAP_CONFIG_PASSWORD: "config"
      LDAP_READONLY_USER: "false"
      #LDAP_READONLY_USER_USERNAME: "readonly"
      #LDAP_READONLY_USER_PASSWORD: "readonly"
      LDAP_RFC2307BIS_SCHEMA: "false"
      LDAP_BACKEND: "mdb"
      LDAP_TLS: "true"
      LDAP_TLS_CRT_FILENAME: "ldap.crt"
      LDAP_TLS_KEY_FILENAME: "ldap.key"
      LDAP_TLS_DH_PARAM_FILENAME: "dhparam.pem"
      LDAP_TLS_CA_CRT_FILENAME: "ca.crt"
      LDAP_TLS_ENFORCE: "false"
      LDAP_TLS_CIPHER_SUITE: "SECURE256:-VERS-SSL3.0"
      LDAP_TLS_VERIFY_CLIENT: "demand"
      LDAP_REPLICATION: "false"
      #LDAP_REPLICATION_CONFIG_SYNCPROV: 'binddn="cn=admin,cn=config" bindmethod=simple credentials="$$LDAP_CONFIG_PASSWORD" searchbase="cn=config" type=refreshAndPersist retry="60 +" timeout=1 starttls=critical'
      #LDAP_REPLICATION_DB_SYNCPROV: 'binddn="cn=admin,$$LDAP_BASE_DN" bindmethod=simple credentials="$$LDAP_ADMIN_PASSWORD" searchbase="$$LDAP_BASE_DN" type=refreshAndPersist interval=00:00:00:10 retry="60 +" timeout=1 starttls=critical'
      #LDAP_REPLICATION_HOSTS: "#PYTHON2BASH:['ldap://ldap.example.org','ldap://ldap2.example.org']"
      KEEP_EXISTING_CONFIG: "false"
      LDAP_REMOVE_CONFIG_AFTER_SETUP: "true"
      LDAP_SSL_HELPER_PREFIX: "ldap"
    tty: true
    stdin_open: true
    command: [--copy-service, --loglevel, debug] #   --loglevel, debug
    volumes:
      - /var/lib/ldap
      - /etc/ldap/slapd.d
      - /container/service/slapd/assets/certs/
      - ./deploy/small.schema:/container/service/slapd/assets/config/bootstrap/schema/small.schema
      - ./deploy/fake.ldif:/container/service/slapd/assets/config/bootstrap/ldif/data.ldif

2. Try running docker-compose up

Workspace affected

No response

Expected behavior

openldap should start without errors.

This should be possible in principle, because when you run it like so (without docker-compose), it seems to work fine:

docker run osixia/openldap:stable

Example repository

No response

Anything else?

Here is a probable cause for this issue.

[MatthewHallCom]

We're seeing this as well with docker-compose inside of Gitpod. It's very inconsistent

[jankeromnes]

@matthewhall-ca Is it the same error for you at the end of the error message? (I.e. write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: read-only file system: unknown)

[MatthewHallCom]

Apologies, no, we're seeing
ERROR: for device-web Cannot start service device-web: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: operation not permitted: unknown

[kylos101]

Hi 👋 , I tried to recreate this issue using the above docker-compose.yml, and it works in both the US and EU cluster (I tried with many workspaces). I even ctrl + c and ran docker-compose up many times, but was unable to recreate the problem.

@jankeromnes is it possible for them to share a public repo from which they are able to reliably recreate the problem?

[jankeromnes]

Many thanks @kylos101 for investigating this issue, and even creating a minimal reproduction case! 🙏 💯

I believe their code is private, so I'm not sure how to effectively make progress here. I'll ask if they can see significant differences between their setup and https://github.com/kylos101/OCI-runtime-create-failed -- but otherwise, I don't see a clear path forward. Maybe a call with some live debugging over screenshare could work, but since this issue isn't currently a blocker for them, I'd be more inclined to pause the investigation for now and wait for more elements (e.g. maybe from someone else who can reproduce the same issue in the future).

[kylos101]

@jankeromnes the other thing I would be curious about is - does this only happen during peak times, or does it happen during non-peak times, too.

I plan to pause the investigation for now, until we get more information.

[konne]

next one, just now, workspace: vizlib-gucci-jgm70a009op

[csweichel]

@konne I reckon that's an issue somewhat different from the one described herein. We just (15:17:00 UTC) rolled out out a (hopefully) fix for the issue you found (see also #5945).

[kylos101]

@gitpod-io/engineering-workspace hey there, I scheduled this issue as its similar to what was solved in #7657.

[kylos101]

next one, just now, workspace: vizlib-gucci-jgm70a009op

Hi @konne I just opened #7666 based on the screenshot you shared. The fix from #7657 was deployed to both the EU and US clusters.

For #7666, and the no such file or directory error, let us know if you're still experiencing, and whether it is intermittent?

[csweichel]

Probably caused by https://github.com/moby/moby/blob/7b9275c0da707b030e62c96b679a976f31f929d3/daemon/oci_utils.go#L13-L14

[csweichel]

next one, just now, workspace: vizlib-gucci-jgm70a009op

Hi @konne I just opened #7666 based on the screenshot you shared. The fix from #7657 was deployed to both the EU and US clusters.

For #7666, and the no such file or directory error, let us know if you're still experiencing, and whether it is intermittent?

The no such file or directory refers to /proc/self/fd which make sense, considering that /proc wasn't mounted at all. Looking at the time, that issue occurred before we had rolled out the fix. It does not look like an issue different from the original one.

We must be careful to not confuse the different issues surrounding Docker container starts. Most of their error messages look similar, which means we really gotta be careful in keeping them apart.
Let's focus on the domainname sysctl in this issue, and if need be, reopen a new one if we see the original proc mount issue reappear now that we've rolled out the fix.

[kylos101]

@matthewhall-ca @jankeromnes hey there, @utam0k produced a fix for this issue in #7705.

It won't be included in today's initial release, but we should be able to include it in a patch soon. More updates to follow!

[kylos101]

Hey @matthewhall-ca and @jankeromnes we released a new generation of clusters including this fix, which should resolve this issue. 💪

I am going to close this issue for now - please let us know if you continue to experience trouble with domainname?

[konne]

@kylos101 just for your info, we had this OCI issue without domainname, but I will check it in the next days.
If this still happens I will ping you