Server Out of Disk

you are here because you got alerted that some server was out of disk.

what you do next depends on which disk is out of space.

df -h will show you which disk is full (if you did not get that from the alert already`

/

This drive is where all the docker images and container data lives. to clean it we need to remove containers and/or old images. The easiest first step is to remove image builder and weave containers

1. Ssh into the server
2. Remove image builder containers docker rm `docker ps -a | awk '/image-builder.*Exited/{print $1}'`
3. Check to see if this helped (you want to maintain atleast 10gb free) df -h | grep docker
4. If that is still not enough you can run khronos manually. see its README.md for instructions on how to manually run for one dock
5. Sometimes we have images created without tags (khronos would clean them). In order to delete them manually try docker images | grep none | awk '{ print $1 }' | while read i; do echo deleting $i; docker rmi $i; done

The root disk is also where the os and logs live. here you can safely delete logs and files in the ~/var/log/ path.

1. Check where the space is getting used the usual suspects du -h -d 0 /var du -h -d 0 /home

2. If /var/log is using too much space first remove archived logs rm /var/log/*.gz

3. If that does not help flush out applications logs 1. echo “” > /var/log/docker-listener.log 2. echo “” > /var/log/charon.log 3. echo “” > /var/log/filibuster.log 4. echo “” > /var/log/krain.log

4. Mark unhealthy with docks cli if all else fails