Core dump of containerd.service on 5:20.10.12~3-0~debian-bullseye_armhf.deb (RasPi ZeroW)

[LorenAmelang]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Installation of
containerd.io_1.4.12-1_armhf.deb 2021-12-11 23:01:38 14.7 MiB
docker-ce-cli_20.10.123-0debian-bullseye_armhf.deb 2021-12-13 14:39:56 34.0 MiB
docker-ce-rootless-extras_20.10.123-0debian-bullseye_armhf.deb 2021-12-13 14:39:58 6.3 MiB
docker-ce_20.10.123-0debian-bullseye_armhf.deb 2021-12-13 14:39:58 12.6 MiB
on Raspberry Pi Zero W with fresh Debian Bullseye

Actual behavior

A dependency job for docker.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com

Jan 23 16:19:43 raspberrypi systemd[1]: Dependency failed for Docker Application Container Engine.
Jan 23 16:19:43 raspberrypi systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u2) ...
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
pi@raspberrypi:~ $

@raspberrypi:~ $ journalctl -xe
░░
░░ A stop job for unit containerd.service has finished.
░░
░░ The job identifier is 143343 and the job result is done.
Jan 23 16:22:41 raspberrypi systemd[1]: Starting containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has begun execution.
░░
░░ The job identifier is 143343.
Jan 23 16:22:41 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, statu>
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit containerd.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 11.
Jan 23 16:22:41 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit containerd.service has entered the 'failed' state with result 'core-dump'.
Jan 23 16:22:41 raspberrypi systemd[1]: Failed to start containerd container runtime.
░░ Subject: A start job for unit containerd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has finished with a failure.
░░
░░ The job identifier is 143343 and the job result is failed.
pi@raspberrypi:~ $

Steps to reproduce the behavior

[https://docs.docker.com/engine/install/debian/] (https://docs.docker.com/engine/install/debian/)

Exploration:

pi@raspberrypi:~ $ ls -al /lib/systemd/system/
total 1028
drwxr-xr-x 21 root root 20480 Jan 23 16:19  .
drwxr-xr-x 17 root root  4096 Dec 16 20:40  ..
...
-rw-r--r--  1 root root  1162 Apr  2  2021  apparmor.service
...
-rw-r--r--  1 root root  1263 Nov 17 10:48  containerd.service
-rw-r--r--  1 root root  1254 Aug  6 20:58  container-getty@.service
...
-rw-r--r--  1 root root  1695 Dec 13 03:44  docker.service
-rw-r--r--  1 root root   175 Dec 13 03:44  docker.socket
...
-rw-r--r--  1 root root   404 Feb  2  2021  sockets.target
drwxr-xr-x  2 root root  4096 Oct 30 04:13  sockets.target.wants


pi@raspberrypi:~ $ sudo find / -name "*containerd.service*"
/usr/lib/systemd/system/containerd.service
/var/lib/systemd/deb-systemd-helper-enabled/containerd.service.dsh-also
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/containerd.service
/etc/systemd/system/multi-user.target.wants/containerd.service
pi@raspberrypi:~ $


pi@raspberrypi:~ $ systemctl status containerd.service
● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: core-dump) since Sun 2022-01-23 16:42:55 PST; 1s ago
       Docs: https://containerd.io
    Process: 6759 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 6760 ExecStart=/usr/bin/containerd (code=dumped, signal=SEGV)
   Main PID: 6760 (code=dumped, signal=SEGV)
        CPU: 366ms

Jan 23 16:42:55 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 23 16:42:55 raspberrypi systemd[1]: Failed to start containerd container runtime.
pi@raspberrypi:~ $

Output of docker version:

pi@raspberrypi:~ $ docker version
Segmentation fault
pi@raspberrypi:~ $

Output of docker info:

pi@raspberrypi:~ $ docker info
Segmentation fault
pi@raspberrypi:~ $

Additional environment details (AWS, VirtualBox, physical, etc.)
This is exactly the 2018 problem reported in:
https://forums.docker.com/t/error-when-starting-docker-service/63041/4

gforghetti
Gary Forghetti
Docker Staff
Feb '19
That is a known issue if you installed Docker 18.09.1.
The workaround is to install 18.06.03 till a patch release becomes available for 18.09.

But I have
docker-ce (5:20.10.123-0debian-bullseye) ...
And it isn't 2019 any more...

#490

thaJeztah commented on Oct 10, 2019
This should be resolved now with the container.io_1.2.10-2 package

But I have
containerd.io is already the newest version (1.4.12-1)

[LorenAmelang]

I gave up on Docker and tried to install my NodeJS project directly, but the Pi was constantly repeating the containerd core dump. I tried every command I could find to disable Docker and containerd, but nothing worked. I finally removed Docker and containerd:

Jan 25 16:57:18 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 25 16:57:18 raspberrypi systemd[1]: Failed to start containerd container runtime.
Jan 25 16:57:23 raspberrypi systemd[1]: containerd.service: Scheduled restart job, restart counter is at 1669.
Jan 25 16:57:24 raspberrypi systemd[1]: Stopped containerd container runtime.
Jan 25 16:57:24 raspberrypi systemd[1]: Starting containerd container runtime...
Jan 25 16:57:24 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, status=11/SEGV
Jan 25 16:57:24 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Jan 25 16:57:24 raspberrypi systemd[1]: Failed to start containerd container runtime.
Jan 25 16:57:29 raspberrypi systemd[1]: containerd.service: Scheduled restart job, restart counter is at 1670.
Jan 25 16:57:30 raspberrypi systemd[1]: Stopped containerd container runtime.
Jan 25 16:57:30 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:33 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jan 25 16:57:35 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:38 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.
Jan 25 16:57:43 raspberrypi systemd[1]: Reloading.
Jan 25 16:57:46 raspberrypi systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted.

Is there some command I didn't find that would have worked to stop the thousands of core dumps?

[Chiller2019]

same here just with

Apr 22 11:14:24 raspberrypi systemd[1]: containerd.service: Main process exited, code=dumped, status=4/ILL
Apr 22 11:14:24 raspberrypi systemd[1]: containerd.service: Failed with result 'core-dump'.
Apr 22 11:14:24 raspberrypi systemd[1]: Failed to start containerd container runtime.

This kernel-version is the problem

[epikhinm]

Same with newer versions of docker-ce and containerd:

schizophrenia@cloudberry:/var/log $ docker ps
Segmentation fault
schizophrenia@cloudberry:/var/log $ dpkg -l | grep container
ii  containerd.io                        1.5.11-1                         armhf        An open and reliable container runtime
iF  docker-ce                            5:20.10.14~3-0~debian-bullseye   armhf        Docker: the open-source application container engine
ii  docker-ce-cli                        5:20.10.14~3-0~debian-bullseye   armhf        Docker CLI: the open-source application container engine

Also on raspbian on zero 1.1:

schizophrenia@cloudberry:~ $ lsb_release -a
uname No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
schizophrenia@cloudberry:~ $ uname -a
Linux cloudberry 5.15.32+ #1538 Thu Mar 31 19:37:58 BST 2022 armv6l GNU/Linux

[epikhinm]

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

[Chiller2019]

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

Because I read in many forums that, when anybody change (downgrade) the kernel-version, its work. I dont know why but it works for me.

[epikhinm]

@Chiller2019

This kernel-version is the problem

Why do you have such guess?

Because I read in many forums that, when anybody change (downgrade) the kernel-version, its work. I dont know why but it works for me.

What the kernel version do you use? Are you use it on raspbian? How do you downgrade it?

[xmmwc]

same issue with raspi zero

[epikhinm]

I switched to podman, as workaround.

[mathmaniac43]

I am having this issue as well.

[dkowis]

Happening for me using DietPi on a Pi Zero W after installing the latest using the instructions from Docker's website.

journalctl -xe

Log Paste

Dec 07 02:47:19 enviro systemd[1]: Starting containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has begun execution.
░░
░░ The job identifier is 43164.
Dec 07 02:47:19 enviro sudo[2019]:   dietpi : TTY=pts/0 ; PWD=/home/dietpi ; USER=root ; COMMAND=/usr/bin/su -Dec 07 02:47:19 enviro sudo[2019]: pam_unix(sudo:session): session opened for user root(uid=0) by dietpi(uid=1000)
Dec 07 02:47:19 enviro su[2020]: (to root) dietpi on pts/0
Dec 07 02:47:19 enviro su[2020]: pam_unix(su-l:session): session opened for user root(uid=0) by dietpi(uid=0)
Dec 07 02:47:20 enviro systemd[1]: containerd.service: Main process exited, code=dumped, status=11/SEGV
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit containerd.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 11.
Dec 07 02:47:20 enviro systemd[1]: containerd.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit containerd.service has entered the 'failed' state with result 'core-dump'.
Dec 07 02:47:20 enviro systemd[1]: Failed to start containerd container runtime.
░░ Subject: A start job for unit containerd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has finished with a failure.
░░
░░ The job identifier is 43164 and the job result is failed.

Packages

root@enviro:~# dpkg -l  | grep docker
iF  docker-ce                     5:20.10.21~3-0~debian-bullseye   armhf        Docker: the open-source application container engine
ii  docker-ce-cli                 5:20.10.21~3-0~debian-bullseye   armhf        Docker CLI: the open-source application container engine
ii  docker-compose-plugin         2.12.2~debian-bullseye           armhf        Docker Compose (V2) plugin for the Docker CLI.
root@enviro:~# dpkg -l | grep containerd
ii  containerd.io                 1.6.10-1                         armhf        An open and reliable container runtime

Kernel Version

root@enviro:~# uname -a
Linux enviro 5.15.76+ #1597 Fri Nov 4 12:11:43 GMT 2022 armv6l GNU/Linux

LSB release info

root@enviro:~# lsb_release  -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

[Kralg]

Hello,

For me this also happens on bullseye_armhf.deb RasPi ZeroW.
A dependency job for docker.service failed.

uname -a
Linux raspberry 6.1.21+ #1642 Mon Apr 3 17:19:14 BST 2023 armv6l GNU/Linux