mingw-w64-cross-clang arm and aarch64 targets #2294

jeremyd2019 · 2020-12-30T22:02:08Z

This PR adds armv7-w64-mingw32 and aarch64-w64-mingw32 targets for mingw-w64-cross-clang{,-headers,-crt} and mingw-w64-cross-compiler-rt.

jeremyd2019 · 2020-12-30T22:36:34Z

This was able to build a test c++ program for armv7 and aarch64, that worked on Windows arm64 (on a raspberry pi)

mingwandroid · 2020-12-31T00:32:57Z

Nice

Biswa96 · 2020-12-31T04:01:06Z

Why add the targets here instead of mingw-w64-clang package? The compiler will not depend on msys2 layer.

jeremyd2019 · 2020-12-31T04:27:08Z

Why add the targets here instead of mingw-w64-clang package? The compiler will not depend on msys2 layer.

I thought of that too. This is the only set of "cross" clang packages I think. I know that clang itself can generate code for all platforms, but the libraries shipped with mingw-w64-clang are only "native", as far as I know you can't even link i686 binaries out-of-the-box with mingw-w64-x86_64-clang (since there is no libclang_rt.builtins-i386.a in its prefix).

Also, I believe new prefixes historically "spring forth" by being cross-compiled from msys2, not from mingw.

Biswa96 · 2020-12-31T04:43:23Z

I believe new prefixes historically "spring forth" by being cross-compiled from msys2

Agreed but we've to compile/port MINGW-packages to aarch64, so I was thinking of adding a version of llvm-mingw in hypothetical /mingwarm64 may help, for example, with makepkg-mingwarm64 command.

This cross compiler will create native aarch64 compiler but that hardware is not available to everyone and the current ones for consumer are slow (except M1).

mati865 · 2020-12-31T14:50:05Z

AFAIK it's not decided yet whether ARM/AArch64 subsystems should be native or cross-compiled. If they end up being cross-compiled then we'd use x86_64 cross Clang for bootstrapping.

jeremyd2019 · 2021-01-01T21:57:59Z

I had a try last night to bootstrap clang on aarch64-w64-mingw32 from the CLANG-packages bootstrap branch... it failed on sanitizer_win.cpp. Well, getting closer anyway

mati865 · 2021-01-05T08:03:11Z

Sanitizers don't build with BFD but should build with LLD, for cross packages they are unnecessary though.

jeremyd2019 · 2021-01-05T20:03:34Z

The problem was actually that sanitizers are not (yet?) supported on Windows arm/aarch64.

mati865 · 2021-01-05T23:43:10Z

Ah right, I thought sanitizers are supported by AArch64 but it might have been not merged yet.

dennisameling · 2021-01-22T21:47:02Z

@jeremyd2019 I have a Surface Pro X (ARM64), feel free to ping my if you'd like me to test anything 👍🏼 heard that Windows 10 on the Raspberry Pi isn't the fastest

On arches other than i686/x86_64, it can be assumed that 0xA00 is a safe minimum _WIN32_WINNT version.

The armv7 lib needs to be called 'arm' not 'armv7' for clang to find it.

jeremyd2019 · 2021-04-11T19:11:57Z

Huh, this used to build. Wonder what happened

jeremyd2019 · 2021-04-11T23:49:07Z

/usr/bin/clang.exe -target aarch64-w64-mingw32 --sysroot=/opt/aarch64-w64-mingw32 -rtlib=compiler-rt  -target aarch64-w64-mingw32 --sysroot=/opt/aarch64-w64-mingw32 -rtlib=compiler-rt -fuse-ld=lld -v -Wl,--whole-archive CMakeFiles/cmTC_500dc.dir/objects.a -Wl,--no-whole-archive -o cmTC_500dc.exe -Wl,--out-implib,libcmTC_500dc.dll.a -Wl,--major-image-version,0,--minor-image-version,0
clang version 11.0.0 (https://github.com/msys2/MSYS2-packages dab5c29f1c61a173e404f1fd78f0c6f56441aa9f)
Target: aarch64-w64-windows-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/bin/ld.lld" --sysroot=/opt/aarch64-w64-mingw32 -m arm64pe -Bdynamic -o cmTC_500dc.exe /opt/aarch64-w64-mingw32/lib/crt2.o /opt/aarch64-w64-mingw32/lib/crtbegin.o -L/opt/aarch64-w64-mingw32/aarch64-w64-mingw32/lib -L/opt/aarch64-w64-mingw32/lib -L/opt/aarch64-w64-mingw32/aarch64-w64-mingw32/sys-root/mingw/lib -L/usr/lib/clang/11.0.0/lib/windows --whole-archive CMakeFiles/cmTC_500dc.dir/objects.a --no-whole-archive --out-implib libcmTC_500dc.dll.a --major-image-version 0 --minor-image-version 0 -lmingw32 /usr/lib/clang/11.0.0/lib/windows/libclang_rt.builtins-aarch64.a -lmoldname -lmingwex -lmsvcrt -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 /usr/lib/clang/11.0.0/lib/windows/libclang_rt.builtins-aarch64.a -lmoldname -lmingwex -lmsvcrt -lkernel32 /opt/aarch64-w64-mingw32/lib/crtend.o
lld-link: error: undefined symbol: __chkstk
>>> referenced by libmingw32.a(libarm64_libmingw32_a-pseudo-reloc.o):(_pei386_runtime_relocator)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [CMakeFiles/cmTC_500dc.dir/build.make:101: cmTC_500dc.exe] Error 1
make[1]: Leaving directory '/e/MSYS2-packages/mingw-w64-cross-clang/src/build-libcxx-shared-aarch64-w64-mingw32/CMakeFiles/CMakeTmp'
make: *** [Makefile:127: cmTC_500dc/fast] Error 2

@mati865 any clue what might be going on? It looks like __chkstk is defined:

/usr/lib/clang/11.0.0/lib/windows/libclang_rt.builtins-aarch64.a(chkstk.S.obj):file format coff-arm64


Disassembly of section .text:

0000000000000000 <__chkstk>:
       0: f0 ed 7c d3   lsl     x16, x15, #4
       4: f1 03 00 91   mov     x17, sp
       8: 31 06 40 d1   sub     x17, x17, #1, lsl #12
       c: 10 06 40 f1   subs    x16, x16, #1, lsl #12
      10: 3f 02 40 f9   ldr     xzr, [x17]
      14: ac ff ff 54   b.gt    0x8 <__chkstk+0x8>
      18: c0 03 5f d6   ret

mati865 · 2021-04-12T00:10:17Z

I have no idea and cannot look into it within next 16 hours.

jeremyd2019 · 2021-04-12T00:10:47Z

OK. I am making some headway I think.

jeremyd2019 · 2021-04-12T00:22:58Z

If mingw-w64-cross-compiler-rt and mingw-w64-cross-clang are built with cmake-3.18.4-1 (the version that was in the repo the last time this worked), it works. If built with cmake-3.20.0-2 (the version that is in the repo now) it does not. It appears that just using the downgraded cmake on cross-clang is not sufficient, compiler-rt must be built with it as well.

I tried some more versions to try to narrow it down.

✔️ cmake-3.18.4-1
✔️ cmake-3.19.3-1
❌ cmake-3.20.0-1
❌ cmake-3.20.0-2
❌ cmake-3.20.1-1

mati865 · 2021-04-13T00:16:49Z

I have no idea, did you try comparing compiler-builtins from working and broken CMake?

jeremyd2019 · 2021-04-13T01:38:03Z

Here is a zip with the working and broken aarch64 compiler-rt libs.
compiler-rt-aarch64-compare.zip

jeremyd2019 · 2021-04-13T17:39:30Z

I had a thought that maybe it was running msys2 ranlib instead of llvm-ranlib, but I just saw a post about how ranlib hasn't actually been needed for years/decades... https://sourceware.org/pipermail/binutils/2021-April/116084.html

jeremyd2019 · 2021-04-13T20:09:00Z

W T F. running llvm-ranlib on the broken one seems to 'fix' it. Well at least I've got a workaround now

mati865 · 2021-04-13T22:12:41Z

Sorry to bother you like this @mstorsjo but I guess you are the only one who can have any idea what is going on in the last few comments (#2294 (comment) and later).
Seems like Cmake regression but maybe it was preexisting LLVM bug?

mstorsjo · 2021-04-14T07:20:59Z

I had a thought that maybe it was running msys2 ranlib instead of llvm-ranlib, but I just saw a post about how ranlib hasn't actually been needed for years/decades... https://sourceware.org/pipermail/binutils/2021-April/116084.html

Well ar does create the index automatically, provided that it understands the objct files being packaged. (Possibly also Binutils ar doesn't recognize the ARM/ARM64 COFF object files so it won't do the right thing there.) Also, ar has got the flag s for doing the indexing automatically when inserting object files - not sure if it is done anyway even if that flag isn't specified. (And having the separate ranlib tool is kinda convenient anyway, exactly for figuring out these cases. IIRC there's some flag to ar to print the symbol index table, so one can inspect whether it's there or not, but I don't remember what the flag was.)

I tried cross-building my toolchain with CMake 3.20.1 (both the current pinned version with LLVM 11, and with a RC of LLVM 12) and it builds fine in both cases, but I configure with -DCMAKE_RANLIB="$PREFIX/bin/llvm-ranlib" too, so there's little risk for mixups.

I'd be interested in seeing a log of the build of compiler-rt with 3.19 vs 3.20 in the case where things break (built with make VERBOSE=1 or ninja -v). I tried comparing my (cross from linux) builds and there was no difference in the command that creates the static library between those two versions, both run llvm-ar qc lib/windows/libclang_rt.builtins-aarch64.a <object files> && llvm-ranlib lib/windows/libclang_rt.builtins-aarch64.a.

Sorry to bother you like this @mstorsjo but I guess you are the only one who can have any idea what is going on in the last few comments

Hey, don't be sorry, I'm happy to help out with people poking these things!

FWIW, I see that you also have the hardcoded hackery for renaming around the arm version of compiler-rt builtins. I've worked on this in the current git main branch (for the 13 release), with that one won't need to hack around with the library names for i386 vs i686 and arm vs armv7 any longer, and just do a ninja install.

mati865 · 2021-04-14T08:20:45Z

Ah, we are using export CC="clang" CXX="clang++" AS="clang" AR="llvm-ar" RANLIB="llvm-ranlib" DLLTOOL="llvm-dlltool" LD="clang" but they are not documented by CMake.

Verbose log sounds like good idea, we should also replace environment variables with -DCMAKE_AR=llvm-ar -DCMAKE_AS="clang" -DCMAKE_C_COMPILER="clang" -DCMAKE_CXX_COMPILER="clang++" -DCMAKE_RANLIB="llvm-ranlib" passed directly to CMake.

jeremyd2019 · 2021-04-14T17:32:43Z

Output of ninja -v building just aarch64. It appears to have been using /usr/bin/ar instead of llvm-ar.

cross-compiler-rt-aarch64-log.txt

jeremyd2019 · 2021-04-14T17:43:56Z

Oddly, CMAKE_AR and CMAKE_RANLIB seem to look in the current working directory for a simple llvm-ar or llm-ranlib. Setting -DCMAKE_AR="$(which llvm-ar)" seems to work though

mstorsjo · 2021-04-14T17:50:20Z

Oddly, CMAKE_AR and CMAKE_RANLIB seem to look in the current working directory for a simple llvm-ar or llm-ranlib. Setting -DCMAKE_AR="$(which llvm-ar)" seems to work though

Yep. CMAKE_C_COMPILER checks the path, CMAKE_AR doesn't - dunno why. (Historical reasons I presume.)

Output of ninja -v building just aarch64. It appears to have been using /usr/bin/ar instead of llvm-ar.

cross-compiler-rt-aarch64-log.txt

Ok, but how did work/differ in 3.19?

jeremyd2019 · 2021-04-14T17:57:06Z

Output of ninja -v building just aarch64. It appears to have been using /usr/bin/ar instead of llvm-ar.
cross-compiler-rt-aarch64-log.txt

Ok, but how did work/differ in 3.19?

Log with cmake-3.19.3-1 msys2 package, not setting CMAKE_AR etc.
cross-compiler-rt-aarch64-cmake-3.19.3-1-log.txt

mstorsjo · 2021-04-14T18:08:40Z

Output of ninja -v building just aarch64. It appears to have been using /usr/bin/ar instead of llvm-ar.
cross-compiler-rt-aarch64-log.txt

Ok, but how did work/differ in 3.19?

Log with cmake-3.19.3-1 msys2 package, not setting CMAKE_AR etc.
cross-compiler-rt-aarch64-cmake-3.19.3-1-log.txt

Ok, so it implicitly chose llvm-ar for some reason, but no longer does - I guess that's understandable.

jeremyd2019 · 2021-04-14T18:21:12Z

Ok, so it implicitly chose llvm-ar for some reason, but no longer does - I guess that's understandable.

Ah, we are using export CC="clang" CXX="clang++" AS="clang" AR="llvm-ar" RANLIB="llvm-ranlib" DLLTOOL="llvm-dlltool" LD="clang" but they are not documented by CMake.

mstorsjo · 2021-04-14T18:32:51Z

Ok, so it implicitly chose llvm-ar for some reason, but no longer does - I guess that's understandable.

Ah, we are using export CC="clang" CXX="clang++" AS="clang" AR="llvm-ar" RANLIB="llvm-ranlib" DLLTOOL="llvm-dlltool" LD="clang" but they are not documented by CMake.

Oh, right, I forgot that bit. RIght, then it makes sense - thanks!

mingw-w64-cross-clang/PKGBUILD

It seems cmake >= 3.20 doesn't respect AR and RANLIB environment variables anymore. Also, cmake seems to use ASM and -DCMAKE_ASM_COMPILER, not AS.

jeremyd2019 · 2021-04-16T17:24:29Z

Is this waiting on anything more from me?

mstorsjo · 2021-04-20T09:45:53Z

Btw, on this topic, how do you call windres for these cross targets? I guess you currently use something like my windres wrapper tool.

I'm working on a proper llvm-windres tool upstream (https://reviews.llvm.org/D100756), which would avoid to use any out-of-tree tools for that purpose. But my question is how you prefer to pass the target arch - in llvm-mingw, I have symlinks (copies) of the binaries with names like <triple>-windres, but I'm wondering if you are going that route here, or if you just are invoking e.g. clang -target <triple> and would like to do the same for windres. With GNU windres, one can choose the target arch of object files with e.g. -F pe-x86-64.

But as GNU binutils don't support COFF-aarch64, there's no canonical BFD target names for that (I guess one could just choose to define pe-arm or pe-armv7, and pe-aarch64 or pe-arm64). In the current state of my patch, I allow users to pass the full triple that way too, e.g. -F aarch64-w64-mingw32. Do you think that's a good way to go, or should we invent BFD target names for this purpose? Which way would you prefer to use it?

mati865 · 2021-04-20T11:16:22Z

Oh, my response was not delivered last time.

Is this waiting on anything more from me?

I think we are just waiting on @lazka.

Btw, on this topic, how do you call windres for these cross targets? I guess you currently use something like my windres wrapper tool.

We are using your exact wrapper:
https://github.com/msys2/CLANG-packages/blob/f26e5b0c2dc557a54421e0a487dfeee1c9302694/mingw-w64-clang/windres-wrapper.c

MSYS2 structure is a bit weird and we don't really have cross compilation. Each target is actually a host so we just call windres which has default target set to ${ARCH}-w64-mingw32.

mstorsjo · 2021-04-20T11:22:02Z

MSYS2 structure is a bit weird and we don't really have cross compilation. Each target is actually a host so we just call windres which has default target set to ${ARCH}-w64-mingw32.

Oh, so you build a wrapper with a hardcoded default target arch via DEFAULT_TARGET? Ok, I see.

Hmm, how would that work for the upstream llvm-windres binary then? Do you build your whole LLVM setup with something like -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-w64-mingw32? That'd work for making the llvm-windres default to the right target too.

mati865 · 2021-04-20T11:38:53Z

Do you build your whole LLVM setup with something like -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-w64-mingw32?

Exactly.

mstorsjo · 2021-04-20T12:09:37Z

Do you build your whole LLVM setup with something like -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-w64-mingw32?

Exactly.

Oh, ok. That should work just fine then.

For other setups - do you happen to have any opinion on the the "BFD format" option to windres, whether to extend it to allow full triples or extend it with new BFD-like format names like "pe-aarch64", or just make up an entirely new syntax for specifying a triple to target? (In particular, being able to write a full triple is kinda useful to let the preprocessor find its sysroot, in case it's not <arch>-w64-mingw32 but some other spelling, without the w64 bit etc.)

jeremyd2019 · 2021-04-20T19:50:18Z

@lazka I don't want to be a pest, but I am waiting on this to merge msys2/CLANG-packages#2 and start bootstrapping 🚀

jeremyd2019 · 2021-04-20T20:16:04Z

thanks!

jeremyd2019 mentioned this pull request Jan 2, 2021

clang: package clang-tblgen.exe #2297

Merged

This was referenced Apr 7, 2021

Arm targets msys2/CLANG-packages#2

Merged

Update mingw-w64 packages to commit 1c7738. #2425

Merged

jeremyd2019 added 4 commits April 11, 2021 10:16

cross-clang-headers: build armv7 and aarch64 targets

05b7372

On arches other than i686/x86_64, it can be assumed that 0xA00 is a safe minimum _WIN32_WINNT version.

cross-clang-crt: build armv7 and aarch64 targets

87d4a75

cross-compiler-rt: build armv7 and aarch64 targets

07090ca

The armv7 lib needs to be called 'arm' not 'armv7' for clang to find it.

cross-clang: build armv7 and aarch64 targets

095b480

jeremyd2019 force-pushed the cross-clang-arm branch from 13875e2 to 095b480 Compare April 11, 2021 17:16

jeremyd2019 closed this Apr 11, 2021

jeremyd2019 reopened this Apr 11, 2021

jeremyd2019 marked this pull request as draft April 11, 2021 19:12

jeremyd2019 force-pushed the cross-clang-arm branch from b88f5bc to 095b480 Compare April 12, 2021 01:21

jeremyd2019 marked this pull request as ready for review April 13, 2021 21:23

jeremyd2019 force-pushed the cross-clang-arm branch from 66e0894 to ba42be4 Compare April 14, 2021 18:05

mati865 reviewed Apr 14, 2021

View reviewed changes

mingw-w64-cross-clang/PKGBUILD Outdated Show resolved Hide resolved

mati865 approved these changes Apr 14, 2021

View reviewed changes

set -DCMAKE_AR, etc in addition to env vars.

f268369

It seems cmake >= 3.20 doesn't respect AR and RANLIB environment variables anymore. Also, cmake seems to use ASM and -DCMAKE_ASM_COMPILER, not AS.

jeremyd2019 force-pushed the cross-clang-arm branch from ba42be4 to f268369 Compare April 14, 2021 19:06

mati865 mentioned this pull request Apr 14, 2021

add gha workflow to build bootstrap packages msys2/CLANG-packages#31

Merged

lazka merged commit 7272347 into msys2:master Apr 20, 2021

jeremyd2019 deleted the cross-clang-arm branch April 20, 2021 20:16

jeremyd2019 mentioned this pull request May 14, 2021

Backport patch for libc++abi msys2/MINGW-packages#8648

Merged

mingw-w64-cross-clang arm and aarch64 targets #2294

mingw-w64-cross-clang arm and aarch64 targets #2294

Conversation

jeremyd2019 commented Dec 30, 2020

jeremyd2019 commented Dec 30, 2020

mingwandroid commented Dec 31, 2020

Biswa96 commented Dec 31, 2020

jeremyd2019 commented Dec 31, 2020

Biswa96 commented Dec 31, 2020 • edited Loading

mati865 commented Dec 31, 2020

jeremyd2019 commented Jan 1, 2021

mati865 commented Jan 5, 2021

jeremyd2019 commented Jan 5, 2021

mati865 commented Jan 5, 2021

dennisameling commented Jan 22, 2021

jeremyd2019 commented Apr 11, 2021

jeremyd2019 commented Apr 11, 2021

mati865 commented Apr 12, 2021

jeremyd2019 commented Apr 12, 2021

jeremyd2019 commented Apr 12, 2021 • edited Loading

mati865 commented Apr 13, 2021

jeremyd2019 commented Apr 13, 2021

jeremyd2019 commented Apr 13, 2021

jeremyd2019 commented Apr 13, 2021

mati865 commented Apr 13, 2021

mstorsjo commented Apr 14, 2021

mati865 commented Apr 14, 2021

jeremyd2019 commented Apr 14, 2021

jeremyd2019 commented Apr 14, 2021

mstorsjo commented Apr 14, 2021

jeremyd2019 commented Apr 14, 2021

mstorsjo commented Apr 14, 2021

jeremyd2019 commented Apr 14, 2021

mstorsjo commented Apr 14, 2021

jeremyd2019 commented Apr 16, 2021

mstorsjo commented Apr 20, 2021

mati865 commented Apr 20, 2021 • edited Loading

mstorsjo commented Apr 20, 2021

mati865 commented Apr 20, 2021

mstorsjo commented Apr 20, 2021

jeremyd2019 commented Apr 20, 2021

jeremyd2019 commented Apr 20, 2021

Biswa96 commented Dec 31, 2020 •

edited

Loading

jeremyd2019 commented Apr 12, 2021 •

edited

Loading

mati865 commented Apr 20, 2021 •

edited

Loading