Support caching cc invocations with -fprofile-use

[michaelwoerister]

At the moment sccache does not try to cache C/C++ compiler invocations with the -fprofile-use flag. However, that would be rather useful for cases where profile data is cached by a build system (instead of regenerated for every build).

It looks like implementing this shouldn't be too hard using the existing ExtraHashFile infrastructure. But one caveat there is that profile data files can become rather big (tens of megabytes are not uncommon). I suspect that re-hashing all that data for each compiler invocation is not very fast.

Does sccache have some kind of mechanism to cache such file hashes (and only re-hash if the file in question has changed)?

cc @luser

[luser]

Does sccache have some kind of mechanism to cache such file hashes (and only re-hash if the file in question has changed)?

It doesn't at present, but #758 is open on the topic. Note that last year the hash algorithm used was switched to BLAKE3, so it should be pretty fast. sccache server logs contain timing information on how long it spends hashing files so it should be easy enough to run some before and after tests if you implement this functionality:

sccache/src/util.rs

Line 141 in b3f4727

"Hashed {} files in {}",

[michaelwoerister]

Yeah, BLAKE3 looks pretty fast. On the other hand compiling (for example) all of LLVM means invoking Clang 2000-3000 times, and for each invocation we would re-hash the same 20MB .profdata file. Although it might not actually take that long on a fast machine, each time sccache hashes ~40-60GB of instead of 20MB you'd not be wrong to expect a sad little tear drop sparkling in the corner of my eye.

From the readme and #758 it sounds like sccache will always spawn a server process, right? That would be the perfect spot for keeping such a cache of file hashes.

Hopefully I'll have time to take a closer like at the whole setup next week.