dockerd fails to start - RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1

[giannello]

After the merge of #461, the docker containers we run as part of our CI jobs stopped working.
The container fails to start with the following error:

failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1

The affected image is docker.io/library/docker@sha256:ae63bb7c7d3ae23884a2c5d206939640279f6d15730618192b58662a0619f182, while docker.io/library/docker@sha256:c90e58d30700470fc59bdaaf802340fd25c1db628756d7bf74e100c566ba9589 works fine. Both images are tagged as 24.0.7-dind

The environment is GKE 1.27 with Container-Optimized OS.

Workaround
Use docker:24.0.7-dind-alpine3.18, as it points at the previous version of the image that was overwritten

[Syphon83]

Hi everyone,
same problem here. Every dind pipeline stopped working.
Every gitlab runner fail with error
Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument

As a quick workaround we had to switch back from dind image docker:24.0.7-dind to docker:23.0.6-dind.
Environment is GKE 1.25.

[giannello]

24.0.6-dind worked for us

[steve-mt]

We (GitLab.com) have a similar issue (https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283) with 24.0.7 failing to start on Google Container Optimized OS.

As we can see with Alpine 3.19 changed the iptables version 👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1695929693

We've also tried multiple Google Container Optmized OS versions and all of them seem to fail 👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1696008058 and also tried some fixes in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1696085805 which didn't work.

The only thing that worked for us is changing the host image to Ubuntu 👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1696057259 but this is not a viable option for us.

---

At the moment I'm not sure what our (GitLab.com) next steps are since Alpine 3.19 seems to be incompatible with Google Container Optimized OS, and it also seems like other users are having the same problem.

cc @tianon

[paolomainardi]

@stevexuereb for us using 24.0.7 with alpine 3.18 fixed the issue.

But of course this is just a temporary workaround.

[steve-mt]

I've tried to use the legacy package instead in the Dockerfile, but still didn't have any luck 👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1696330857

[dgteixeira]

This is also happening for us when using ARC for GitHub runners.

Since it used dind:latest it broke our self-hosted runners pipeline.

We will try to force a previous version as mentioned above and will give feedback back asap.

EDIT (27/12/2023):
This is how we, temporarily, solved it using the ARC helm chart.

We added this to our helm chart:

image:
  repository: "summerwind/actions-runner-controller"
  actionsRunnerRepositoryAndTag: "summerwind/actions-runner:latest"
  dindSidecarRepositoryAndTag: "docker:24.0.7-dind-alpine3.18"

[jnoordsij]

I've tried to use the legacy package instead in the Dockerfile, but still didn't have any luck 👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17283#note_1696330857

Not exactly sure on all the moving bits involved here, but given the changeset in #461, you might want to revert the change to the dockerd-entrypoint.sh file in there as well, to ensure the modprobe loads ip_tables rather than nf_tables.

[akerouanton]

@stevexuereb I'm not sure why your change doesn't fail to build (since that RUN command starts with set -eu), but your ln command is wrong. You need to pass it the -f flag to overwrite the existing symlink.

I'm gonna open a PR.

[steve-mt]

@stevexuereb I'm not sure why your change doesn't fail to build (since that RUN command starts with set -eu), but your ln command is wrong. You need to pass it the -f flag to overwrite the existing symlink.

I'm gonna open a PR.

@akerouanton interesting, it doesn't seem to have existing links, I wasn't installing the iptables apk.

I've tested your PR and it seems to be working:

$ gcloud compute instances create docker-test-cos-85 --image cos-85-13310-1498-7 --zone=us-east1-c

$ gcloud compute ssh docker-test-cos-85

steve@docker-test-cos-85 ~ $ git clone https://github.com/akerouanton/docker-doi.git

steve@docker-test-cos-85 ~/docker-doi $ git switch fix-nf_tables
steve@docker-test-cos-85 ~/docker-doi $ cd 24/dind
steve@docker-test-cos-85 ~/docker-doi/24/dind $ docker build -t sxuereb:find .
steve@docker-test-cos-85 ~/docker-doi/24/dind $ docker run --rm --privileged sxuereb:dind
Certificate request self-signature ok
subject=CN = docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN = docker:dind client
/certs/client/cert.pem: OK
ip: can't find device 'nf_tables'
modprobe: can't change directory to '/lib/modules': No such file or directory
ip: can't find device 'ip_tables'
...
time="2023-12-15T13:49:31.883884128Z" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2 version=24.0.7
time="2023-12-15T13:49:31.884533827Z" level=info msg="Daemon has completed initialization"
time="2023-12-15T13:49:31.931701127Z" level=info msg="API listen on /var/run/docker.sock"
time="2023-12-15T13:49:31.932209555Z" level=info msg="API listen on [::]:2376"