[Bug]: Windows LLVM binaries compile extremely slowly

[AndrewAtAvenza]

Description

There seems to be something wrong with the Windows LLVM binaries (specifically, clang.exe, clang++.exe, maybe the other clang executables).

I posted a comparison of timings from builds on my machine in the discussion forum (#1967). The easiest way to replicate this is to build anything substantial, an excellent case being curl.

1. Download curl 8.4.0 from https://curl.se/download/curl-8.4.0.tar.gz
2. Extract to a folder
3. Open a command window and move to the root of the extracted folder
4. Set the following variables:

SET ANDROID_SDK_HOME=C:\Users\[username]\AppData\Local\Android\Sdk

SET CMAKE_HOME=%ANDROID_SDK_HOME%\cmake\3.22.1

SET ABI_TYPE=arm64-v8a

SET NDK_VERSION=26.1.10909125

In the above example, I used Android Studio to install the NDK & CMake, but they could come from anywhere

5. Run:

%CMAKE_DIR%\bin\cmake -B builds\Debug\%ABI_TYPE% -G Ninja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=.\out\Debug\%ABI_TYPE% -DCMAKE_TOOLCHAIN_FILE=%ANDROID_SDK_HOME%\ndk\%NDK_VERSION%\build\cmake\android.toolchain.cmake -DCMAKE_MAKE_PROGRAM=%CMAKE_DIR%\bin\ninja.exe -DANDROID_PLATFORM=android-26 -DANDROID_ABI=%ABI_TYPE% -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTING=OFF -DBUILD_CURL_EXE=OFF -DENABLE_ARES=OFF -DCURL_DISABLE_TELNET=ON -DCURL_DISABLE_DICT=ON -DCURL_DISABLE_TFTP=ON -DCURL_DISABLE_LDAP=ON -DCURL_DISABLE_LDAPS=ON -DCURL_DISABLE_RTSP=ON -DCURL_DISABLE_POP3=ON -DCURL_DISABLE_IMAP=ON -DCURL_DISABLE_SMTP=ON -DCURL_DISABLE_GOPHER=ON -DCURL_DISABLE_MQTT=ON -DCURL_ENABLE_SSL=OFF

6. After that, run:

%CMAKE_DIR%\bin\cmake --build builds\Debug\%ABI_TYPE% --config Debug --target install

The CMake step should take a long while. The build step (6) should also be pretty obviously slow. If you set the NDK version to 25.0.8775105 and repeat the steps, you should see a significant difference in speed.

As a side note, a friendly user on the LLVM forum suggested I try replacing the clang.exe & clang++.exe from the 26.1.10909125 LLVM prebuilt folder with the ones from the official LLVM 17.0.2 release, so I gave that a try. When I did that, the build time was similar or faster than with 25.0.8775105 -- so the problem seems to lie with the LLVM windows binaries that ship with 26.1.10909125.

Upstream bug

No response

Commit to cherry-pick

No response

Affected versions

r26

Canary version

No response

Host OS

Windows

Host OS version

10.0.19045.3570

Affected ABIs

armeabi-v7a, arm64-v8a, x86, x86_64

[yabinc]

I confirmed that compiling curl using ndk r25 takes 6 seconds, whiling using ndk r26 takes 3 minutes.
Since the official LLVM 17 release seems to have no problem, I will check if this is caused by any local patch or local build step.

[yabinc]

I am not sure if I reproduced the same problem.
On my windows machine, the time difference seems all caused by the load time of clang.exe.
I tried time running clang --version:
For clang-r450784c, it takes about 1s.
For clang-r450784d, it takes about 0.06s.
For clang-r450784d1 (in ndk r25), it takes about 0.06s.
For clang-r450784e, it takes about 1s.
For clang-r487747d (in ndk r26), it takes about 1s.
For upstream clang 17.0.1 and 17.0.2 release, it takes about 1s.

And the time for compiling a c file in libcurl is similar to the time loading clang.exe.

So on my windows machine, only clang-r450784d and clang-r450784d1 are loaded significantly faster than other versions.
It doesn't match the situation in the bug, that "with the ones from the official LLVM 17.0.2 release, so I gave that a try. When I did that, the build time was similar or faster than with 25.0.8775105".

I feel it may be something universal, or something special on my machine (like executable firewall).
@AndrewAtAvenza , can you help check the time of clang --version of different clang.exe on your machine?

[AndrewAtAvenza]

I'm not sure if I did this right, but I used this command via git bash to time the executions:

time ./clang++ --version

When I ran it for the r26.1.10909125 build, I got this this the first time I ran it:

real    0m2.057s
user    0m0.000s
sys     0m0.625s

The second time, and every time after that though, I got times similar to this:

real    0m0.138s
user    0m0.000s
sys     0m0.015s

I tried this in the r26.1.10909125 folder with the 17.0.2 I downloaded straight from LVVM and I got basically the same result; a slow execution, then always fast after that.

I went to the 25.0.8775105 folder and tried it there, but always got times like this:

real    0m0.033s
user    0m0.000s
sys     0m0.000s

This happened even on the first execution. So it does seem to be behaving differently, but it's not clear why. It's also odd because this seems to suggest that it caches the executable after the first execution, speeding it up after that. Assuming my timing test is legit!

If there's a better way to perform the timing let me know and I'll try it.

[yabinc]

I finally get over the execution cache problem.
After restarting my windows machine, it can cache clang.exe in ndk26.
$ time android-ndk-r26b/toolchains/llvm/prebuilt/windows-x86_64/bin/clang --version
Android (10552028, based on r487747d) clang version 17.0.2 (https://android.googlesource.com/toolchain/llvm-project d9f89f4d16663d5012e
5c09495f3b30ece3d2362)
Target: x86_64-w64-windows-gnu
Thread model: posix
InstalledDir: C:/Users/yabinc/bugs/clang_slow/android-ndk-r26b/toolchains/llvm/prebuilt/windows-x86_64/bin

real 0m0.215s
user 0m0.000s
sys 0m0.062s

===================================================

After that, I can finish compiling curl using ndk26 within 7 seconds.

C:\Users\yabinc\bugs\clang_slow\curl-8.4.0>echo start time 15:32:39.66
start time 15:32:39.66
C:\Users\yabinc\bugs\clang_slow\curl-8.4.0>C:\Users\yabinc\AppData\Local\Android\Sdk\cmake\3.22.1\bin\cmake --build builds\Debug\arm64-v8a-ndk26 --config Debug
[165/165] Linking C static library lib\libcurl-d.a
C:\Users\yabinc\bugs\clang_slow\curl-8.4.0>echo end time 15:32:46.91
end time 15:32:46.91

So I can't reproduce your slow compilation problem. Maybe you can try restarting your machine, or try another windows machine?

[AndrewAtAvenza]

@yabinc

My machine has been rebooted a few times since then; I'll give it a try there and then find another Windows machine to try it on as well.

[AndrewAtAvenza]

@yabinc I'm seeing the same result you are now, it's building curl as fast as NDK v25. I'm sorry to have wasted your time here!

Any idea what caused this? It sounds like it was somehow not refusing to cache the clang++.exe until a reboot was performed. Is that a known requirement on Windows? That seems a bit bizarre to me.

The main thing is the problem isn't happening though -- thanks for looking at this for me!

[yabinc]

I have the same guess: Windows refused to cache clang.exe in ndk r26 before reboot.
Since the problem disappeared after reboot, it's also hard to reproduce. Probably not worth further effort here.
I found something similar online:
https://groups.google.com/a/chromium.org/g/chromium-dev/c/PA08hu52-v8/m/rprWBLlcBAAJ

The problem is also slow compiler launch time. And the fix they had was adding exclusive rules for Windows Defender.