Skip to content

Commit

Permalink
Check free disk space before starting Docker (#2097)
Browse files Browse the repository at this point in the history
It seems that Docker can fail to start if there is no space left on the
device. Try to free up some space in that case by asking journald to
limit its size to 256MiB.

This should work for any storage larger than ~2.5GiB (as the journals
maximum size is 10% of the disk size). It still should leave enough
logs to diagnose problems if necessary.

Note: We could also limit the size of the journal in first place, but
that isn't sustainable: Once that space is used up, we run into the
same problem again.

By only asking journalctl to free up if necessary, we kinda (miss)use
the journal as way to "reserve" some space which we can free up at boot
if necessary.
  • Loading branch information
agners authored Aug 31, 2022
1 parent ba5de20 commit ed554f2
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
FailureAction=reboot

[Service]
ExecStartPre=/usr/libexec/docker-disk-check
ExecStopPost=/usr/libexec/docker-failure
12 changes: 12 additions & 0 deletions buildroot-external/rootfs-overlay/usr/libexec/docker-disk-check
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/sh

# Check if less than 128MiB is available on /mnt/data (4k block size).
if [ "$(stat -f /mnt/data -c '%f')" -lt 32768 ]; then
echo "The system is very low on disk space!"
echo "This can cause Docker to fail to start, causing a boot loop."
echo "Asking systemd-journald to free up some space."
# systemd-journald defaults to 10% of disk size. So this should free up
# space for any system which has been running for a while (presumably it
# has when there is no space) and has a disk size larger than ~2.5GiB.
journalctl --vacuum-size=256M
fi

0 comments on commit ed554f2

Please sign in to comment.