Disk full errors should be spurious

[Noratrieb]

https://crater-reports.s3.amazonaws.com/pr-115235/try%235a3d3b91048a0adf280e7a4e589c1dda6443f172/gh/SapientAsh.solstice_calculator/log.txt

This is a spurious error but isn't marked as such.

[INFO] [stdout]   = note: /usr/bin/ld: final link failed: No space left on device
[INFO] [stdout]           collect2: error: ld returned 1 exit status

[RalfJung]

https://crater-reports.s3.amazonaws.com/pr-116284/index.html also has quite a few of these.

[tbu-]

Also happened here: https://crater.rust-lang.org/ex/pr-124636.

[Skgland]

Also happened here: https://crater.rust-lang.org/ex/pr-124636.

I see 213 entries correctly sorted under spurious-regressed with build no space left on device.
And 9 or 10 under regressed with build failed (unknown)

For the 10 the errors are

• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/d0392f4632b22de6528ebe4efcc10f002656b8723c4a1b710ed4458e6b02ea3e-init: no space left on device
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/bd2885f2ca702c73f2089fd560ca5812020f20a703dd920f909fcf68b043099e/diff: no space left on device
• [INFO] [stderr] Error response from daemon: symlink ../1744f2c6d6c1b7751a30adc89afb973fac6960b6b240c59be797348b8d713855/diff /var/lib/docker/overlay2/l/YQMKEEA6H4GSSXCHI2Y5LO3XBH: no space left on device
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/0fa118698fba3bdf766053a429fda884e20458aa114597c09bd675a92e77adcb-init: no space left on device
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/35445b29951a97ad96f344bd33abb3c7976dcd0a7db9620eb7ebfaed05284b94-init: no space left on device
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/5cd8254e4074e5a5dc108cf93c21d54d17667b05d9d70dd70a7404d4a2600f55-init: no space left on device
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/d1f525505d6a64c02a2fc42e1e7e0fa163877b4776a8c7ca3c94d69358ca848e-init: no space left on device
• [INFO] [stderr] Error response from daemon: write /var/lib/docker/containers/08cdd65fbdc00b34e781e9ffe505de039f45012ef9e480a41d13c7b4780739d5/.tmp-config.v2.json2346282875: no space left on device
• log truncated, no error show, so uncertain if this is even a no space left on device error
• [INFO] [stderr] Error response from daemon: mkdir /var/lib/docker/overlay2/9404567f07eaeadab3dc07a00c6d4ea16806738d6eaa4b83b6ed78c401868307-init: no space left on device

So it looks like for all these the docker daemon failed to perform some action due to the full device.
Another source not seen here is described in #715. i.e. when a dependency fails to compile it is always a DependsOn failure even if a no space left on device error occurred.
This can be seen in crater result posted above by RalphJung under regressed: dependencies

https://crater-reports.s3.amazonaws.com/pr-116284/index.html also has quite a few of these.

[tbu-]

What leads to the disk being full for crater?

[Skgland]

To reduce computation time crater does not reset to a pristine environment between crates so that the builds of dependencies can be reused. So build artifacts accumulate and can fill up the disk.

There is supposed to be a disk space every now and then see this comment https://github.com/rust-lang/crater/blob/master/src/runner/tasks.rs#L172-L185 that is supposed to cleanup when that happens.
The disc-space-watcher is started over here https://github.com/rust-lang/crater/blob/master/src/runner/mod.rs#L98 and appear to be currently configured to run every 30 seconds with a threshold of 80%.

[tbu-]

I see, thank you.

So in a sense, disk-full errors are spurious, but shouldn't be ignored, but and the crates maybe retried after cleaning up some disk space.