CLS crash: Microsoft.VisualStudio.Setup.Configuration.Native.dll only has arm64ec build

[saschanaz]

I have arm64 clang build and it seems CLS tries to load arm64ec build of Microsoft.VisualStudio.Setup.Configuration.Native.dll and crashes. Checking C:\ProgramData\Microsoft\VisualStudio\Setup\ only shows arm64ec and x86.

Logs
E[23:07:57.167] Indexing D:\gecko\accessible\windows\ia2\ia2AccessibleTextSelectionContainer.cpp failed: Couldn't build compiler instance
I[23:07:57.167] --> $/progress
I[23:07:57.167] --> $/progress
#0 0x00007ffd5ffafd50 (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x7fd50)
#1 0x00007ffd5ff40af4 (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x10af4)
#2 0x00007ffd5ff3499c (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x499c)
#3 0x00007ffd5ff42360 (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x12360)
#4 0x00007ffd5ff436dc (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x136dc)
#5 0x00007ffd5ff437dc (C:\ProgramData\Microsoft\VisualStudio\Setup\arm64ec\Microsoft.VisualStudio.Setup.Configuration.Native.dll+0x137dc)
#6 0x00007ff6bd04e4a8 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x1a9e4a8)
#7 0x00007ff6bce88430 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x18d8430)
#8 0x00007ff6bcb41028 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x1591028)
#9 0x00007ff6bcb46608 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x1596608)
#10 0x00007ff6bc3a2b48 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0xdf2b48)
#11 0x00007ff6bbd4b624 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x79b624)
#12 0x00007ff6bcc6958c (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x16b958c)
#13 0x00007ff6bcc6ba70 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x16bba70)
#14 0x00007ff6bbd48d90 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x798d90)
#15 0x00007ff6bcc6a9b4 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x16ba9b4)
#16 0x00007ff6bbdf2af8 (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x842af8)
#17 0x00007ff6bd0f6b9c (C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin\clangd.exe+0x1b46b9c)
#18 0x7b177ffdf84a86f0

System information
Clangd version (from the log, or clangd --version): 18.1.7
clangd extension version: 0.1.29
Operating system: Windows 11 arm64

[HighCommander4]

What is CLS?

[saschanaz]

Clang Language Server.

[HighCommander4]

Thanks. That said, I don't think I quite understand the issue well enough to figure out the next steps.

From the stack trace, clangd is calling into a Visual Studio dll. Is that expected? Could you try running the stack trace through llvm-symbolizer to see where in the code clangd is calling into this dll? And is there a way to get more information about why the crash inside the dll itself happens?

[saschanaz]

From the stack trace, clangd is calling into a Visual Studio dll. Is that expected?

Honestly don't know, I was assuming that it's something bundled with clangd extension... But I cannot find any mention of Microsoft.VisualStudio.Setup.Configuration.Native.dll her nor in llvm-project repo. 🤔

Could you try running the stack trace through llvm-symbolizer to see where in the code clangd is calling into this dll?

Can you elaborate how I can do this?

And is there a way to get more information about why the crash inside the dll itself happens?

Probably because mixing arm64 and arm64ec is by design not supported because of different ABI.

[HighCommander4]

Where did you get your clangd binary from -- did you build it yourself, or from one of the binary packages at https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.7 or somewhere else?

Could you try running the stack trace through llvm-symbolizer to see where in the code clangd is calling into this dll?

Can you elaborate how I can do this?

An LLVM release package should contain an llvm-symbolizer binary, in the same bin directory as the clangd binary. (Or if you built clangd yourself, you can build this binary as well with ninja llvm-symbolizer or similar.) You can pipe a stack trace through it (e.g. cat stacktrace.txt | llvm-symbolizer), and assuming the addresses in the stack trace come from the same binaries as llvm-symbolizer, it will add symbol names.

And is there a way to get more information about why the crash inside the dll itself happens?

Probably because mixing arm64 and arm64ec is by design not supported because of different ABI.

I see. So maybe this is an issue with how clangd was built, or in the LLVM build system (that it links to the wrong system libraries)?

[saschanaz]

llvm-symbolizer gives the same mangled name _GUID_9fc77a4b_3c1c_44ed_a798_6c1deea53e1f for all the addresses in the above log. 😞

I'm building Firefox and clangd is provided via bootstrap:

• Binary: https://treeherder.mozilla.org/jobs?repo=toolchains&searchStr=clang-tidy&selectedTaskRun=DWMzZfjgQZC4vjzJbym90A.0
• Build script (to call cmake): https://searchfox.org/mozilla-central/rev/5f641898e12dafdfb7d54a93f4f3164d2c1642b2/build/build-clang/build-clang.py

Clang tools there are being cross compiled from x86-64 Linux. Perhaps some missing build parameter? 🤔

[saschanaz]

Or how did it end up having only arm64ec and x86 in VS directory. that may be more important question...

[HighCommander4]

LLVM does have CI for this platform (https://lab.llvm.org/buildbot/#/builders/161), so it should be possible to get it to work in principle.

[HighCommander4]

Maybe a helpful direction could be to compare the CMake command line used in LLVM CI, which can be seen here, to the one used by the build-clang.py script.

[HighCommander4]

(Though note that the LLVM CI is not cross-compiling, it's actually building on an arm64 windows host.)

[saschanaz]

Any idea where clangd would possibly load VS dll though?

[HighCommander4]

Any idea where clangd would possibly load VS dll though?

I don't, sorry.

But one thing that would be interesting to try is using other binaries in C:\Users\sasch.mozbuild\clang-tools\clang-tidy\bin (I'm assuming there is at least clang-tidy), to see if they have the same issue or not.

[saschanaz]

I still only have x86 and arm64ec binary of that dll on my machine but somehow the crash doesn't happen anymore. I'll come back when it happens again.