Replace sccache with ccache for WoA bots

[omjavaid]

ccache is now available on Windows on Arm machines and this
patch will enable to use to ccache instead of sccache on all
Windows on Arm buildbots hosted by Linaro.

[asl]

@omjavaid What are benefits of ccache compared to sccache?

[compnerd]

My experience sccache has been more stable than ccache where I've seen some miscompilations due to incorrect cache handling, so I'm also curious about the answer to aloe's question.

[omjavaid]

@asl @compnerd The exact reason for making this change is to evaluate whether ccache is better or worse on WoA platform. Both sccache and ccache run in emulation mode in our Surface X Windows on Arm machines and We have seen sccache server hanging on the these bots in the past. We can always revert back to sccache if ccache is not stable or worse.

[sylvestre]

Is there a bug on the sccache side ?

Miscompilation with sccache are very unlikely given how it works. It is probably an issue elsewhere in the stack.
It has been used for a very long time for Firefox.

(Disclosure: i am one of the maintainer of sccache)

[omjavaid]

Most of the time its connection sccache server timing out. Similar to this
https://lab.llvm.org/buildbot/#/builders/120/builds/5827/steps/4/logs/stdio. Problem resolves only after we kill the existing sccache server.

[omjavaid]

Other than the issue of sccache server timing out, there is possibility that Given both sccache and ccache run in emulation mode on windows on Arm we might actually have ccache performing slightly better.

[sylvestre]

I opened this issue to have a better error message mozilla/sccache#1938

Given both sccache and ccache run in emulation mode on windows on Arm we might actually have ccache performing slightly better.

I am curious, why do you think that? :)

[omjavaid]

I opened this issue to have a better error message mozilla/sccache#1938

Given both sccache and ccache run in emulation mode on windows on Arm we might actually have ccache performing slightly better.

I am curious, why do you think that? :)

I dont have concrete numbers to support this but we have seen some apps perform better than others when run in Windows on Arm emulation mode.

[compnerd]

I'm curious why sccache would run in emulation mode. Rust does support arm64, so we should be able to get a native binary?

@sylvestre if that was in response to my message, I'm saying that ccache has issues, not sccache :). Now only if sccache supported Swift 😛

[omjavaid]

@asl @compnerd After above discussion I have gone back and tried to pull some crude compilation time stats using both sccache and ccache on Windows.

@sylvestre Fix for 1730 has broken sccache clang-cl + MSVC build on Windows. I have filed mozilla/sccache#1953

Note: Benchmarks were run with local cache. Both ccache (version 0.5.1) and sccache (4.8.2) were installed via scoop without any additional configurations.

x64 Windows (Intel 13900K + 32GB RAM + Samsung 980 Pro SSD) Room Temperature 25 C

LLVM Configuration

-DCMAKE_BUILD_TYPE=Release ^
-DLLVM_ENABLE_ASSERTIONS=True ^
-DCMAKE_TRY_COMPILE_CONFIGURATION=Release ^
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;llvm;lld" ^
-DCLANG_DEFAULT_LINKER=lld ^

Average Compilation time for first compilation with cache cleared was ~15 minutes with both ccache and sccache.

Re-build with no change in source code:

• ccache ~60 seconds
• sccache ~110 seconds

AArch64 Windows MS Surface X (SQ2 Processor + 16GB RAM + 500 GB SSD) Room Temperature 25 C

LLVM Configuration

-DCMAKE_BUILD_TYPE=Release ^
-DLLVM_ENABLE_ASSERTIONS=True ^
-DCMAKE_TRY_COMPILE_CONFIGURATION=Release ^
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;llvm;lld" ^
-DCOMPILER_RT_BUILD_SANITIZERS=OFF ^
-DLLVM_TARGETS_TO_BUILD=AArch64 ^
-DCLANG_DEFAULT_LINKER=lld ^

Average Compilation time for first compilation with cache cleared was more than 1 hour with both ccache and sccache.

Re-build with no change in source code:

• ccache ~7 Minutes
• sccache ~11 Minutes

[sylvestre]

@omjavaid sorry but your benchmark is lacking information.
is it a local build with local cache or using remote caching (ie gcp or s3)?

[omjavaid]

@sylvestre Benchmarks were run with local cache. Both ccache and sccache were installed via scoop without any additional configurations.

[sylvestre]

ok, it is out of scope for this bug.
But local != distant.

ccache had a feature that sccache didn't have - direct mode:
https://ccache.dev/manual/3.7.9.html#_the_direct_mode

it explained the gap between the two projects.
now, sccache has this feature too.

if you try with the latest release of sccache (0.7.1) and set the variable SCCACHE_DIRECT=true before starting the sccache server, you should get closer performances.

[omjavaid]

@sylvestre sccache latest version 0.7.1 is not usable with clang-cl +MSVC due to the bug mozilla/sccache#1953

[sylvestre]

@omjavaid ok, which version did you use for your benchmark?

[omjavaid]

@omjavaid I have used 0.5.1 for the benchmark.

[gkistanova]

Hi Omair,
Do you still want this change after the discussion?

[omjavaid]

@gkistanova yes this needs to merged because apparently ccache is performing better than sccache's last usable version 0.5.1 on Surface X Pro.

[sylvestre]

I don't think it should be merged yet: sccache 0.7.2 should be as fast as ccache and the issues mentioned earlier should be fixed in 0.7.2.

[omjavaid]

@sylvestre we Linaro Ltd maintain these buildbots and we want to see if any performance improvements can be observed in our build times with changing over to ccache. We can always revert back to sccache or any other better alternative.

[sylvestre]

Sure, makes sense but could you please to update sccache 0.7.2 here? At least to check if the errors you reported have been fixed
https://crates.io/crates/sccache/0.7.2

[omjavaid]

@sylvestre I have tried sccache version 0.7.2 on Windows x86_64 with both cl and clang-cl compilers. The "/wd4668" issue seems to be fixed now. However I was not able to successfully compile LLVM with sccache v0.7.2. The build got stuck compiling at various points and doesnt seem to continue. I have not investigated the issue with v0.7.2 but the same LLVM build just works out of the box with sccache v0.5.1.

[omjavaid]

@gkistanova ping!!

[sylvestre]

@omjavaid

The build got stuck compiling at various points and doesnt seem to continue.

We had the same issue with Firefox

I think we fixed it in 0.7.3 by reverting the faulty patch. It would be nice if you could try again.

[omjavaid]

@sylvestre I have tested sccache 0.7.3 building LLVM on Windows x64 with following configuration flags:

-DCMAKE_BUILD_TYPE=Release ^
-DLLVM_ENABLE_ASSERTIONS=True ^
-DCMAKE_TRY_COMPILE_CONFIGURATION=Release ^
-DLLVM_ENABLE_PROJECTS="clang;lld" ^
-DLLDB_TEST_USER_ARGS="--skip-category=watchpoint" ^
-DCMAKE_C_COMPILER_LAUNCHER="sccache" ^
-DCMAKE_CXX_COMPILER_LAUNCHER="sccache" ^
-DCLANG_DEFAULT_LINKER=lld ^

SCCACHE
Clean build with cache cleared took 12m 30s
Re-built took 1m 45s
NOTE: set SCCACHE_DIRECT=true causes build to freeze as before.

CCACHE
Clean build with cache cleared took 12m 30s
Re-built took 1m 5s

[sylvestre]

@omjavaid excellent, thanks! We are working on a fix for SCCACHE_DIRECT=true

[omjavaid]

@gkistanova Ping! The requested change is harmless from the perspective of LLVM buildbot ecosystem and concerns bots hosted by us (Linaro Ltd). If you have no objections Kindly accept the patch review so that I can merge it and request a master restart.

[gkistanova]

LGTM.