Skip to content

Behemoths periodically run out of inodes #1633

Closed
@lrytz

Description

@lrytz

Spinning off the discussion from scala/scala-dev#732 (comment) into a new ticket

Indeed it looks like inodes is more likely the issue than actual disk space. On behemoth-1:

admin@ip-172-31-2-3:~$ df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvdj         25M   25M  816K   97% /home/jenkins

while disk space looks fine

/dev/xvdj       393G  244G  130G  66% /home/jenkins

The community build workspaces have huge numbers of files and directories. For example, for "scala-2.13.x-jdk11-integrate-community-build" there are currently 103 extraction directories

admin@ip-172-31-2-3:~$ ls /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction | wc -l
103

A single one of those has > 200k inodes:

admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 | wc -l
207593

Looking at things a bit, it seems we could save > 40% of inodes by not pulling in all the git refs to pull requests. They look similar to this:

/home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658/projects/93deaed81507c97b97bdf01b44a6723b14827dc1/.git/refs/pull/110

Some directory counting

admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type d | wc -l
80892
admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type d | grep '\/pull\/' | wc -l
43463

Looking at files in the extraction, again a large number of git refs corresponding to pull requests

admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type f | wc -l
126693
admin@ip-172-31-2-3:~$ find /home/jenkins/workspace/scala-2.13.x-jdk11-integrate-community-build/target-0.9.17/extraction/82d4b745facaabd414be8c97dd8725a670038658 -type f | grep -E 'pull\/[0-9]+\/head' | wc -l
43463

@SethTisue do you think we can do something about these git refs to pull requests?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions