Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raspberry crashes on docker-compose up #7051

Closed
RSWilli opened this issue Nov 23, 2019 · 17 comments
Closed

Raspberry crashes on docker-compose up #7051

RSWilli opened this issue Nov 23, 2019 · 17 comments

Comments

@RSWilli
Copy link

RSWilli commented Nov 23, 2019

Description of the issue

I am running Manjaro ARM on my raspberry PI 3B. I tried to start the homeassistant/raspberry3-homassistant image from docker-compose

It did not work, and the PI froze (needed a hard reset to work again)

Context information (for bug reports)

The issue was, that i tried to start a arm32 image on my arm64 pi, and Manjaro lacks the support for arm32

Running the image with docker gave this error: standard_init_linux.go:211: exec user process caused "exec format error" (which is totally user-unfriendly btw)

docker-compose straight up crashed the whole system instead

Output of docker-compose version

docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.0
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Output of docker version

Client:
 Version:           19.03.5-ce
 API version:       1.40
 Go version:        go1.13.4
 Git commit:        633a0ea838
 Built:             Sat Nov 16 00:08:25 2019
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.5-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       633a0ea838
  Built:            Sat Nov 16 00:07:54 2019
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.3.0.m
  GitCommit:        d50db0a42053864a270f648048f9a8b4f24eced3.m
 runc:
  Version:          1.0.0-rc9
  GitCommit:        d736ef14f0288d6993a1845745d6756cfc9ddd5a
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config

services:
  homeassistant:
    container_name: home-assistant
    environment:
      TZ: Europe/Berlin
    image: homeassistant/raspberrypi3-homeassistant:stable
    ports:
    - published: 8123
      target: 8123
    restart: always
    volumes:
    - /home/whale/hassio/config:/config:rw
version: '3.7'

Steps to reproduce the issue

  1. Raspberry PI with Manjaro ARM (Maybe other distros work as well)
  2. docker run --name="home-assistant" -e "TZ=Europe/Berlin" -v $PWD/config:/config -p 8123:8123 homeassistant/raspberrypi3-homeassistant:latest
    and docker-compose up with the given config
  3. observe the difference ;)

Observed result

docker-compose crashes the system, while docker gives an error

Expected result

docker-compose gives an error

Stacktrace / full error message

for docker:

standard_init_linux.go:211: exec user process caused "exec format error"

docker-compose:
no error, since Pi crashes

Additional information

OS version / distribution, docker-compose install method, etc.
Linux raspi 5.3.12-1-MANJARO-ARM #1 SMP Wed Nov 20 16:23:31 UTC 2019 aarch64 GNU/Linux

docker-compose installed through yay -S docker-compose

@ndeloof
Copy link
Contributor

ndeloof commented Nov 25, 2019

Note: at time writting, ARM is not an officially suported plaform for docker-compose (even we are aware many users do rely on it).

The issue you describe is pretty weird, as docker-compose do fully rely on the docker engine to run containers by it's HTTP API, and as such could fail with ugly errors, but I harldy understand how a python process can crash the system this way.
Can you please try to run the same docker-compose file with -d (dettached) parameter ? Doing so compose will ask engine to run container in a "fire an forget" style, which might limit interactions with the engine and help understand what's causing this issue

@RSWilli
Copy link
Author

RSWilli commented Nov 29, 2019

I tried that, but the shell never came back

I was connected via SSH to my Pi, and after docker-compose up (-d) the shell never came back and the SSH session timed out
A reconnect was only possible after i un- and replugged the powercable from the Pi. Meanwhile my router didn't show my Pi as connected to the network

@danc
Copy link

danc commented Feb 15, 2020

Hi !
Exact same problem on an olimex pine64

uname -m
aarch64

docker-compose up -d
standard_init_linux.go:211: exec user process caused "exec format error"

@ypicard
Copy link

ypicard commented Apr 2, 2020

Same here, docker-compose up crashes my raspberry pi 4 with Raspbian Buster Lite

uname -m
armv7l

@leifle
Copy link

leifle commented Aug 21, 2020

Just to add something to those that might look for the reason of this. In my case it happens almost 100% of the time.
The Pi completely dies on me in that moment. Not even an entry in the syslog.
Now the real problem starts. When I turn on the pi again, I get 5 or so pings and it is dead again. I have to assume this is exactly because the containers come back up and the same thing crashes.
So there is as good as no way to get back onto the pi. Only turning on and off until it works is the way.

My path took me in the strangest corners:
For a while I thought it is the problem with the HDMI frequency problem and the network adapters not working.
Also I had "success" with putting the pi in the fridge as I thought maybe the electronics is too hot.
The latest things is the stupid MAC randomization. Whomever had that idea and how long it takes to find out that this might be a reason. If one tests hardware like this obviously one boots "a lot". So the routers device list in the DHCP runs out of space.

Bottom line
There is something wrong in the docker-compose up (-d) function that makes it crash. Not all the time. So after 10 or more boots I am fine. But that can not be. Usually I don't copy syslog to anybody, but maybe it helps the developers. You can see when I executed docker-compose up -d this happened:

Aug 21 16:14:13 raspberrypi NetworkManager[400]: <info>  [1598019253.7421] device (vethb774c34): released from master device br-ee59a112e310
Aug 21 16:14:13 raspberrypi systemd[1]: run-docker-netns-afeff46e4915.mount: Succeeded.
Aug 21 16:14:13 raspberrypi systemd[629]: run-docker-netns-afeff46e4915.mount: Succeeded.
Aug 21 16:14:13 raspberrypi systemd[1]: var-lib-docker-containers-02b57ec546052fbe88f9ebc810ba37a49157ba676bc752c8e633cab99c88fdb5-mounts-shm.mount: Succeeded.
Aug 21 16:14:13 raspberrypi systemd[629]: var-lib-docker-containers-02b57ec546052fbe88f9ebc810ba37a49157ba676bc752c8e633cab99c88fdb5-mounts-shm.mount: Succeeded.
Aug 21 16:14:13 raspberrypi systemd[1]: var-lib-docker-overlay2-2fd767b8a8025133e1fe4154f7c22966702fcfb25a4e2d9082e10541aa439fcb-merged.mount: Succeeded.
Aug 21 16:14:13 raspberrypi systemd[629]: var-lib-docker-overlay2-2fd767b8a8025133e1fe4154f7c22966702fcfb25a4e2d9082e10541aa439fcb-merged.mount: Succeeded.
Aug 21 16:14:25 raspberrypi systemd[1]: run-docker-runtime\x2drunc-moby-b2a7a9664f86b0f5be0697083874c65cd3bfc8a492e1b8bc6f8b6b06d3876dcb-runc.ehI0YR.mount: Succeeded.
Aug 21 16:14:25 raspberrypi systemd[629]: run-docker-runtime\x2drunc-moby-b2a7a9664f86b0f5be0697083874c65cd3bfc8a492e1b8bc6f8b6b06d3876dcb-runc.ehI0YR.mount: Succeeded.
Aug 21 16:14:29 raspberrypi systemd[1]: run-docker-runtime\x2drunc-moby-23263b0ba8578ef3376693e00df807ce2bbedcf064c842391003ece11e6553ad-runc.akcmnN.mount: Succeeded.
Aug 21 16:14:29 raspberrypi systemd[629]: run-docker-runtime\x2drunc-moby-23263b0ba8578ef3376693e00df807ce2bbedcf064c842391003ece11e6553ad-runc.akcmnN.mount: Succeeded.
Aug 21 15:54:41 raspberrypi systemd-modules-load[117]: Inserted module 'i2c_dev'
Aug 21 15:54:41 raspberrypi fake-hwclock[114]: Fri 21 Aug 2020 12:13:19 PM UTC
Aug 21 15:54:41 raspberrypi systemd-fsck[128]: e2fsck 1.44.5 (15-Dec-2018)
Aug 21 15:54:41 raspberrypi systemd-fsck[128]: root: clean, 284639/1929536 files, 2967855/7712256 blocks

@tam481
Copy link

tam481 commented Aug 31, 2020

Hi all,
I'm getting a similar thing with my RPi 4. It only happens with multiple docker-compose.yml files. If I start a single project with multiple docker images it works fine but if I start another project with multiple images from a docker-compose.yml file it works fine until I shutdown or reboot then it starts up, pings for a few seconds and I can SSH in but then shortly afterwards it freezes.

I'm running docker-compose 1.26.2

@leifle
Copy link

leifle commented Sep 3, 2020

Looks to me like it is not really too widely a problem as few write. Shortly after my post I just created a new SD with Raspian again, the version with GUI and few applications.

Everything works fine with multiple restarts and updates when I did this to install docker and docker-compose. The steps before this are just update things and zsh.

        6)
            echo "----- Install docker"
            sudo apt install -y docker.io
            sudo systemctl enable --now docker
            docker --version
            docker run hello-world
            ;;
        7)
            echo "----- Install docker-compose"
            sudo apt install -y docker-compose
            docker-compose version
            ;;
        8)
            echo "Add current user to docker group"
            sudo usermod -aG docker ${USER}
            ;;

I presume it is caused by something else I have running or an experiment that left stuff behind.

Just so somebody doing a pure docker installation knows that it is easy. Just new install and run the few lines above.

Good luck ...

@tam481
Copy link

tam481 commented Sep 21, 2020

Hi
I tried the steps above but it's still not working. No matter what I do, if I reboot or shutdown and start the Raspberry Pi, it comes up for a few seconds and then stops responding.

I tried the latest release 1.27.3. It doesn't crash, the containers start fine the Pi itself dies after a reboot. Otherwise, it carries on working fine if left powered on

@leifle
Copy link

leifle commented Sep 22, 2020

Hi tam481!

Here I give you the whole script that I run right after a new raspberry install. After this I clone one more git with my docer compose thing and that is the whole thing.

What I experienced with docker and obviously with docker-compose is that sometimes the containers that are latest have issues. Somehow they don't work right for a few days. If it is the manifest or outright the wrong container in the sense that a x86 binary is in somewhere but the Pi is ARM. Try to chose older versions of the container where you know that they proved themselves.
Due to this problem one of my next projects is to write some scripts that revert back to older containers and I will go also further and put them on my own repository. Reason being that dockerhub is overflowing with data and they can not provide this for free anymore. So they just announced a few weeks ago that they will delete images which are not or rarely used. So I think the best is if you safe the functioning images, however you do it. I for my part will setup a docker environment on a cloud server where I have a git repository already so I am not dependent on company decissions by git or dockerhub.

I removed some things that are specific to my setup and files that you don't have.

If you have a completely naked installation of Raspian, this should work. Besides the hello-world you have portainer which gives you at least some GUI. Maybe you can see something there. Also pulling images of different versions is possibly easier for you. At least it is in some cases for me.

Else I can only wish you the best of luck. Mine is still working fine and when I update my images everything starts up fine as well.

#!/bin/bash
# Load package dialog if it is not installed
pkgs='dialog'
if ! dpkg -s $pkgs >/dev/null 2>&1; then
sudo apt-get install -y $pkgs
fi

# Dialog stuff
cmd=(dialog --separate-output --checklist "Select options:" 22 76 16)
options=(1 "Update" on    # any option can be set to default to "on"
         2 "Upgrade" on
         3 "Full Upgrade" off
         4 "Install GIT" off
         5 "Install ZSH (Oh-My-ZSH)" off
         6 "Install Docker" off
         7 "Install Docker-Compose" off
         8 "Add current user to docker group" off
         9 "Start Portainer in Docker" off
         10 "STILL TO COME!! - Start Portainer in Docker-Compose Stack SYS with others" off
         11 "Add .bash_aliases function" off
         12 "Install Python3 and PIP3" off
         20 "REBOOT WITHOUT ASKING!!!" off)
choices=$("${cmd[@]}" "${options[@]}" 2>&1 >/dev/tty)
clear
for choice in $choices
do
    case $choice in
        1)
            sudo apt-get -y update
            ;;
        2)
            sudo apt-get -y upgrade
            ;;
        3)
            sudo apt-get -y full-upgrade
            ;;
        4)
            sudo apt install -y git
            git --version
            ;;
        5)
            # bash SUB/ZSH/first-inst-zsh.sh
            ;;
        6)
            echo "----- Install docker"
            sudo apt install -y docker.io
            sudo systemctl enable --now docker
            docker --version
            docker run hello-world
            ;;
        7)
            echo "----- Install docker-compose"
            sudo apt install -y docker-compose
            docker-compose version
            ;;
        8)
            echo "Add current user to docker group"
            sudo usermod -aG docker ${USER}
            ;;
        9)
            echo "----- Portainer"
            echo "----- Create Volume for Portainer"
            docker volume create portainer_data
            echo "----- Run Portainer"
            docker run -d -p 9000:9000 -p 8000:8000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
            ;;
        10)
            echo "The function to mount stack SYS is still to come!"
            ;;
        11)            
            # This is to make it global for bash. You need to enter your user password. Works only if you are a sudoer.
            ;;
        12)
            sudo apt install -y python3-pip
            pip3 --version
            echo "As for the installation of Python3 you should already have it installed with the OS"
            echo "If not use: sudo apt-get install python3.6 or whatever is current."
            ;;
        20)
            sudo reboot now
            ;;
     esac
done

Here you have my docker-compose.yml. It will not work like this for you because you are missing the environment files with the passwords and so on, but just so you see what I load. If you load something similar I can change some of the environment files.

version: '2'
services:

  portainer:
    container_name: portainer
    image: portainer/portainer-ce
    restart: unless-stopped
    ports:
      - 9000:9000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./volumes/portainer/data:/data

  nodered:
    container_name: nodered
    build: ./services/nodered/.
    restart: unless-stopped
    user: "0"
    privileged: true
    env_file: ./services/nodered/nodered.env
    ports:
      - 1880:1880
    volumes:
      - ./volumes/nodered/data:/data

  influxdb:
    container_name: influxdb
    image: "influxdb:latest"
    restart: unless-stopped
    ports:
      - 8086:8086
      - 8083:8083
      - 2003:2003
    env_file:
      - ./services/influxdb/influxdb.env
    volumes:
      - ./volumes/influxdb/data:/var/lib/influxdb
      - ./backups/influxdb/db:/var/lib/influxdb/backup

  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    restart: unless-stopped
    user: "0"
    ports:
      - 3000:3000
    env_file:
      - ./services/grafana/grafana.env
    volumes:
      - ./volumes/grafana/data:/var/lib/grafana
      - ./volumes/grafana/log:/var/log/grafana

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto
    restart: unless-stopped
    user: "1883"
    ports:
      - 1883:1883
      - 9001:9001
    volumes:
      - ./volumes/mosquitto/data:/mosquitto/data
      - ./volumes/mosquitto/log:/mosquitto/log
      - ./services/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf

  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "8089:80/tcp"
      #- "443:443/tcp"
    env_file:
      - ./services/pihole/pihole.env
    volumes:
       - ./volumes/pihole/etc-pihole/:/etc/pihole/
       - ./volumes/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/
    dns:
      - 127.0.0.1
      - 1.1.1.1
    # Recommended but not required (DHCP needs NET_ADMIN)
    #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

  tradfri-mqtt:
    container_name: tradfri-mqtt
    image: hemtjanst/tradfri-mqtt:arm7
    env_file:
      - ./services/tradfri-mqtt/tradfri-mqtt.env
    volumes:
      - ./volumes/tradfri-mqtt/data:/app/data
    restart: unless-stopped

By the way this is a slightly changed version of what you can make out of IOThub.

@matt1
Copy link

matt1 commented Jan 17, 2021

I've also observed this on a RPi 4 - connected via SSH, run docker-compose up and the SSH dies after a second or two and cant reconnect. Doesn't happen every time - perhaps 1 in 5 or 1 in 10 for me.

I believe that the machine is not crashing, but actually that the wlan0 interface is somehow getting removed so obviously the SSH dies and cannot come back up.

I was able to reproduce the issue via SSH over Wifi while I also had HDMI connected and a local keyboard and was able to get the output of ifconfig after the issue reproduced (i.e. I could login locally after SSH had died over wifi), and could see that wlan0 was now missing from ifconfig, and there were now a number of veth***** interfaces. I had a look in dmesg but there was no obvious smoking gun (indeed no mention of wlan0 at all), although there was what appeared to be a lot of flapping, e.g. something like this (or slight variations of it) was repeated many times within a couple of seconds after running docker-compose up:

[  315.665255] eth0: renamed from veth5d6485f
[  315.692373] IPv6: ADDRCONF(NETDEV_CHANGE): veth37655a6: link becomes ready
[  315.692481] br-8d6e192eae56: port 7(veth37655a6) entered blocking state
[  315.692488] br-8d6e192eae56: port 7(veth37655a6) entered forwarding state
[  315.827072] eth0: renamed from veth86dcd60
[  315.846535] br-8d6e192eae56: port 6(veth65224e7) entered disabled state
[  315.918498] br-8d6e192eae56: port 5(veth9b76cc0) entered disabled state
[  315.998754] br-8d6e192eae56: port 6(veth65224e7) entered blocking state
[  315.998775] br-8d6e192eae56: port 6(veth65224e7) entered forwarding state
[  315.999080] br-8d6e192eae56: port 4(veth93f5bdf) entered disabled state

So obviously since I could still use the RPi locally, it had not crashed entirely - it appears just the wifi network interface is deleted somehow by docker or docker-compose. I am running a totally ordinary local network using consumer router (Fritzbox if that matters) The RPi 4 has a static DHCP assignment on the 192.168.1.x subnet.

Please advise if there is any more diagnosis steps that might help here.

Versions etc:

$ uname -a
Linux raspberrypi 5.10.4-v8+ #1389 SMP PREEMPT Wed Jan 6 13:52:18 GMT 2021 aarch64 GNU/Linux

$ docker --version
Docker version 20.10.2, build 2291f61

$ docker-compose --version
docker-compose version 1.27.4, build unknown

@matt1
Copy link

matt1 commented Jan 23, 2021

Prior to my previous comment about this appearing to be an issue related to the wlan0 network interface disappearing when docker-compose up is called, it would appear that this is related to DHCP and the veth* interfaces.

This thread on the RPi forums provided a fix: https://www.raspberrypi.org/forums/viewtopic.php?t=282425#p1712939

Add this to /etc/dhcpcd.conf and then reboot:

denyinterfaces veth*

So far, with this applied, I have not been able to reproduce the issue at all - I can start and stop docker-compose as many times as I want now without any issues. I tested perhaps 20 or 30 times toggling up and down without any issue.

@mxmaxime
Copy link

mxmaxime commented Feb 2, 2021

Hi,

I had the exact same issue as @matt1 and his solution works like a charm. Thanks!

@chrisn-au
Copy link

As it appears for me - So many thanks were going crazy

@williamdes
Copy link

Hi @aiordache @ulyssessouza
Are contributions welcome to enable multi arch builds on Docker image ?

I maintain https://hub.docker.com/r/botsudo/action-docker-compose and would like to distribute multi arch images

@M0hammedImran
Copy link

I tried that, but the shell never came back

I was connected via SSH to my Pi, and after docker-compose up (-d) the shell never came back and the SSH session timed out
A reconnect was only possible after i un- and replugged the powercable from the Pi. Meanwhile my router didn't show my Pi as connected to the network

This happened to on an AWS instance with 50 GB disk and 2 GB ram

@stale
Copy link

stale bot commented Apr 16, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 16, 2022
@stale
Copy link

stale bot commented Apr 27, 2022

This issue has been automatically closed because it had not recent activity during the stale period.

@stale stale bot closed this as completed Apr 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests