Windows x64 and Ubuntu CI jobs run out of space

[kunaltyagi]

There have been CI failures for Windows x64 check. Both #3386 and #3354 have the same issue

Current Behavior

d:\a\1\s\surface\include\pcl\surface\impl\surfel_smoothing.hpp(84): fatal error C1083: Cannot open compiler intermediate file: 'pcl_surface.dir\Release\mls.obj': Not enough space [D:\a\build\surface\pcl_surface.vcxproj]
LINK : fatal error LNK1257: code generation failed [D:\a\build\surface\pcl_surface.vcxproj]

[taketwo]

I believe I've seen this before on multiple occasions, but not too often. Any ideas about what to do with this? @SunBlack you are into VS, are there any tricks to reduce memory requirements?

[SunBlack]

My current machine has 40GB, so it is not relevant for me ;-). But as far as I remember I had memory issue (swapping) on my old laptop - and there it helped to turn off /MP. I didn't evened lost compiling time, because with less compiler task he didn't needed anymore to swap data from RAM to HDD. But I don't believe swapping is an issue on Azure.

But in general: clcache could help, as he don't need to compile this much anymore. But not sure in case this exception occurs, if clcache caches this exception 🤔

[kunaltyagi]

caches this exception

I don't think ccache will store the exception since it seems to be storing {key: file|compiler|options, value: obj-output}. With an error, there's nothing to store

[taketwo]

Hm, with ccache enabled, we run out of space in Ubuntu builds as well (for example, this one). I found the following passage on the Microsoft website:

It seems like free tier limitations can be increased on a per-project basis. I'm not sure if it only applies to the number of jobs, or the data volume as well. But we may try to contact them.

[taketwo]

I've created a support ticket about this. Let's see what they answer.

[taketwo]

Unfortunately, the storage space limit can not be lifted:

We do not have the ability to increase the free disk space limit on hosted machines. Everyone in free tier and paid tier get the same free disk space.

However, we got 2 extra parallel jobs as a consolation prize:

Regarding the parallel jobs, they were able to change the free public MS hosted value from 10 to 12 for https://dev.azure.com/PointCloudLibrary

[kunaltyagi]

the storage space limit can not be lifted

That's a bit of a sad news. I'm still plodding away at trying to reduce cache size on #3415.

For Windows, I fail to see the benefit of a compiler cache in that case since we are exceeding the space limit already... (Unless we delete unneeded files while building which sounds like a hassle)

2 extra parallel jobs

This can offset waiting time due to the new pipeline for format checking 😄

[taketwo]

I'm still plodding away at trying to reduce cache size

Thanks. We need that!

This can offset waiting time due to the new pipeline for format checking

Well, that pipeline only takes 1 minute anyway. But actually, I've been planning to add another Ubuntu job to the matrix.

[kunaltyagi]

Ubuntu job to the matrix

18.04 hopefully!!

[taketwo]

I'd rather go for the most-up-to-date version, i.e. 19.04 right now (and 19.10 soon). This would allow us to catch issues with new versions of CMake, compilers, dependency packages before they get "frozen" into the next LTS.

[stale]

Marking this as stale due to 30 days of inactivity. It will be closed in 7 days if no further activity occurs.

[kunaltyagi]

Instead of Release, we could try using MinSizeRel to reduce the binary size and save space on the disk

[SunBlack]

I think we should better take a look on our generated code, why it is so big:

And yeah, the release lib is 500MB!!, during the debug is only 36MB. Don't ask me why 😳

[traversaro]

I am not sure about Azure Pipelines, but for GitHub Actions (that share some infrastructure with Azure Pipelines) a workaround I have successfully used is to move the build directory to C: instead of the default drive D:.

[kunaltyagi]

That's 500KB, right? 😕 Still huge compared to debug version.

Any takers for my bet? 1 or more of these 3:

1. Aggressive inlining
2. Loop unrolling
3. Run-time compiled alternates

move the build directory

BUILD_REPOSITORY_LOCALPATH is indeed D:. Thanks for the suggestion. Will look into it over the weekend.

[SunBlack]

That's 500KB, right? 😕 Still huge compared to debug version.

Nop, the . is the delimiter for thousands. On current master: 530 MB (556.575.332 bytes) release vs. 34,9 MB (36.698.124 bytes) debug (see screenshot is a few days old)

[traversaro]

move the build directory

BUILD_REPOSITORY_LOCALPATH is indeed D:. Thanks for the suggestion. Will look into it over the weekend.

At least on GitHub Actions the actual available size depend on the image version, so tipically I monitor the actual available space via a step that executes on bash df -h to monitor the actual available space on the system, see:

• https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/blob/v0.3.0/.github/workflows/ci.yml#L18
• https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/runs/652161005#step:3:4

[kunaltyagi]

Thanks @traversaro That's a nifty trick.

Created 2 issues and a PR. Is there anything else we can do for this?

[mvieth]

@kunaltyagi I would say this issue can be closed? I believe I haven't seen a CI job running out of space since we switched the Windows CI to Docker.

[kunaltyagi]

Yeah. Docker also saved a lot of time, and more coverage for windows