dind-rootless: undeterministic behavior for "long" running processes (cgroup deleted: unknown)

[jappenzesr]

Issue Description
It seems that "long" running processes end most of the time with "docker: Error response from daemon: cgroups: cgroup deleted: unknown.". I identified the issue by trying to run an nginx container, and manage to boil it down to various runs of the sleep command "docker run -it alpine sleep n"

/ # docker run alpine sleep 0.00001
/ # docker run alpine sleep 0.0001
/ # docker run alpine sleep 0.001
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
ERRO[0000] error waiting for container: context canceled
/ # docker run alpine sleep 0.001
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
ERRO[0000] error waiting for container: context canceled
/ # docker run alpine sleep 0.001
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
ERRO[0000] error waiting for container: context canceled
/ # docker run alpine sleep 0.001
/ # 

Note that the last run of the sleep command with the same argument executed successfully.
The same error message is observed when trying to run nginx as daemon, or ubuntu in interactive mode.

Context
We are running a large-scale build platform (based on Atlassian Bamboo) with containerized build agents run on top of kubernetes. Docker build pipelines could be easily enabled by running a dind sidecar container in the same pod as the main agent. However, the dind sidecar container needs to be executed in privileged security context which leads to security issues (as an example). In this context, rootless dind would lead to a significant improvement.

Setup
Container A running 19.03.1-dind-rootless as dind-server.

docker run --privileged --name dind-server -d \
    --network some-network --network-alias docker \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v certs:/certs \
    docker:19.03.1-dind-rootless --experimental

Container B running 19.03.01-dind as dind-client.

docker run --name dind-client --network some-network \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v certs:/certs:ro \
    -it docker:19.03 /bin/sh

Host: Ubuntu 16.04.6 , Docker CE 18.09.6

Commands are executed in dind-client.

/ # docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89e8a
 Built:             Thu Jul 25 21:17:37 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89e8a
  Built:            Thu Jul 25 21:27:55 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
/ #

Server Log
log.txt

[wglambert]

The same error message is observed when trying to run nginx as daemon, or ubuntu in interactive mode.

So you get the same Error response from daemon: cgroups: cgroup deleted: unknown. from just a docker run -d nginx or a docker run -it ubuntu?

This seems like an issue either with the host's configuration or something with the underlying Docker engine

Users running Alpine on their host encountered the same error when starting Docker with --rootless
moby/moby#38817

And this issue had the same error
moby/moby#39234

[Caligatio]

I've been trying to figure this out for the last 12 hours without any luck using a CentOS 7 host with a hand-built 19.03.1 CentOS 7-based dind image. While those other issues certainly appear to be related, they don't really describe what was wrong.

Just as a sanity check, I ran rootless Docker on the host (not dind) without any troubles. cgroups seem to be the common issue here but they're definitely not my strong suite.

[AkihiroSuda]

The issue should have been fixed in containerd/containerd#3419 (backport to v1.2: containerd/containerd#3428), but containerd still doesn't have a proper release with these commits

[jappenzesr]

Issue still present in 19.03.2, @AkihiroSuda any idea when containerd will be released with this fix?

[AkihiroSuda]

in a few days containerd/containerd#3534

[tianon]

Given that this isn't really an issue with this image (and is really more an issue with the software we've packaged), I'm going to close.
(To put it another way, there's not something we're going to change in this image to fix this -- it's going to be fixed by an updated containerd in an updated Docker release.)

For folks that want to mess with / test this further, I'd suggest trying to replace the containerd that's embedded in this image with the now-released 1.2.8.