Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check free disk space before starting Docker (#2092) #2097

Merged

Conversation

agners
Copy link
Member

@agners agners commented Aug 31, 2022

It seems that Docker can fail to start if there is no space left on the
device. Try to free up some space in that case by asking journald to
limit its size to 256MiB.

This should work for any storage larger than ~2.5GiB (as the journals
maximum size is 10% of the disk size). It still should leave enough
logs to diagnose problems if necessary.

Note: We could also limit the size of the journal in first place, but
that isn't sustainable: Once that space is used up, we run into the
same problem again.

By only asking journalctl to free up if necessary, we kinda (miss)use
the journal as way to "reserve" some space which we can free up at boot
if necessary.

It seems that Docker can fail to start if there is no space left on the
device. Try to free up some space in that case by asking journald to
limit its size to 256MiB.

This should work for any storage larger than ~2.5GiB (as the journals
maximum size is 10% of the disk size). It still should leave enough
logs to diagnose problems if necessary.

Note: We could also limit the size of the journal in first place, but
that isn't sustainable: Once that space is used up, we run into the
same problem again.

By only asking journalctl to free up if necessary, we kinda (miss)use
the journal as way to "reserve" some space which we can free up at boot
if necessary.
@agners
Copy link
Member Author

agners commented Aug 31, 2022

This is a bit crude, but it seems to work in my testing. The regular root reserve unfortunately doesn't work as most services run in containers as root. Better would be to use actual disk quota, and enforcement. But disk quotas are a pain to manage, and there is overhead associated to it.

An alternative to the current solution would be to store a "reserve" file, which we can delete at startup. But that is rather crude too, and leads to unnecessary writes (as the file needs to really be written out, a sparse file would not do the job here).

@agners agners merged commit ed554f2 into home-assistant:dev Aug 31, 2022
@agners agners mentioned this pull request Sep 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants