[error] No space left on device

[xerxesb]

Hi,

I’ve created an action workflow that builds our docker image for distribution. This image requires Tensorflow, CUDA, and a number of other libs so it’s reasonably weighty coming in at about 18GB.

During a build run, my workflow fails part way through citing “##[error]No space left on device”

I think I read somewhere that Github runners have about 16GB of space, so my question is
a) whether there’s a way to increase that size limit?
b) How do other teams manage this problem?

Happy to explore alternative solutions to this…Just not sure why there seems to be very little content when I search for this specific issue.

Thanks.

[otter-computer]

Hi @xerxesb , welcome to the GitHub Support Community! Yeah, GitHub-hosted runners only have 14GB of free space available, and there isn’t a way to increase that I’m afraid.

docs.github.com

Specifications for GitHub-hosted runners - GitHub Docs

//docs.github.com/en/actions/reference/specifications-for-github-hosted-runners

The best solution I can think of if you need more space or a more powerful machine would be to host your own runner:

docs.github.com

About self-hosted runners - GitHub Docs

//docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

You should be able to host this where you want—on a local machine, a VM in the cloud—and doing this should give you the flexibility to configure the specs you need for your workflows.

[solodon4]

You don’t need to go hosted runner approach. You can free enough space for a docker image by using this script from Apache Flink: free_disk_space.sh.

I am starting with 48GB available after running it, just make sure to also add monodoc-http before mono-devel in it as the script is probably a bit outdated.

The bigger problem I see is that when the error in subj does happen, no logs are available for the step that failed so there is no way to investigate why you actually ran over allocated space. GitHub should at least preserve the on screen logs of the step as I regularly output data there to let me figure what exactly eat up all the space, but without that log, I’m just poking at lines of code in the dark

[dnut]

While this general approach could work in theory, the script you've linked is not useful for github workflow runners, since it's clearly written for a different system. It removes less than 1 GB, and most of the things it attempts to remove are not even on the runner. This script needs significant modification to be useful. I instead solved this by just removing a single folder, /opt/hostedtoolcache. You can find a detailed explanation in my other post in the current thread: #discussioncomment-5242449

[shw1n]

Thank you @dnut -- I just added rm -rf /opt/hostedtoolcache and my problem was solved

This works!

[xerxesb]

That’s fantastic! I ran it on our repo and free space went from 19% to 53% (I also removed some other packages in the top 100 that weren’t required for our use)

Thanks. This has definitely helped!

[ChesterZheng]

Where could I run the shell free_disk_space.sh?

[xerxesb]

You run it in your workflow. I have it defined as one of the first steps in a job

[dnut]

I freed 10 GB by deleting /opt/hostedtoolcache. Nothing in this folder is necessary if your workflow runs in docker. I also had to switch from using the workflow "container" configuration, to instead configuring the individual steps to run inside the container with the poorly documented uses: docker:// syntax, because the step that deletes /opt/hostedtoolcache needs to run outside of the container.

  foo:
    runs-on: ubuntu-latest
    steps:
      - name: Delete huge unnecessary tools folder
        run: rm -rf /opt/hostedtoolcache

      - uses: actions/checkout@v3

      - name: Run foo
        uses: docker://foo-image
        with: 
          args: /foo-script

FYI these are the contents of the folder:

/opt/hostedtoolcache
├── CodeQL
│   ├── 2.12.2-20230207
│   └── 2.12.3-20230217
├── Java_Temurin-Hotspot_jdk
│   ├── 11.0.18-10
│   ├── 17.0.6-10
│   └── 8.0.362-9
├── PyPy
│   ├── 3.7.13
│   ├── 3.8.16
│   └── 3.9.16
├── Python
│   ├── 3.10.10
│   ├── 3.11.2
│   ├── 3.7.15
│   ├── 3.8.16
│   └── 3.9.16
├── Ruby
│   └── 3.1.3
├── go
│   ├── 1.17.13
│   ├── 1.18.10
│   ├── 1.19.6
│   └── 1.20.1
└── node
    ├── 14.21.3
    ├── 16.19.1
    └── 18.14.2

[loyal812]

It working well. Thanks. <3

[ravife]

Awesome! Worked like a charm.

[glesperance]

This is awesome -- got me unstuck on building my cuda images <3

[ryanhill1]

This worked like a charm, thanks!

[aguizaro]

Thanks for this!

[josephcopenhaver]

Went with this successfully and reclaimed over 12g:

cd /opt
find . -maxdepth 1 -mindepth 1 '!' -path ./containerd '!' -path ./actionarchivecache '!' -path ./runner '!' -path ./runner-cache -exec rm -rf '{}' ';'

Figured I'd keep the 4 items in the list as they had potential for my purposes.

You can always spin up a tmate server in an action and hop into the runner to inspect what's going on in the land of denmark: https://github.com/josephcopenhaver/gha-debian-tmate

[glesperance]

^ This one got rid of even more unused files in our case. 🙌