Clang/LLVM 13 and PE exported symbol limits

[jeremyd2019]

While preparing the update to mingw-w64-clang (really LLVM) 13.0.0, it failed to link libLLVM-13.dll due to:

lld-link: error: too many exported symbols (max 65535)

This limit is due to the PE/COFF format using a 16-bit integer as in index into the export tables (IIUC). There are only a couple of options to get around this:

1. Switch to static linking of libLLVM. It's been a while, but there was a bug that required this to be done here previously, and I recall it making VERY large packages. I have a CI run going with static linking and will update with the size difference when it's done.
2. Disable enough things to get the number of exports back down under 65535. I was able to do this by only enabling targets that I felt might be useful on Windows: AArch64;AMDGPU;ARM;NVPTX;WebAssembly;X86.

Either of these options is probably going to negatively impact somebody, so I thought it would be a good idea to open a discussion to see if there's a consensus on what would be the least-bad option here, and to see if there are any suggestions to adjust the list of targets for option 2.

Package size comparison

(the net upgrade size is from 12.0.1, and most of that on the dynamically-linked side is from the new package "mlir"). Unfortunately the other MSYSTEMs hit the 6 hour GHA limit while packaging/compressing.

MINGW64

Statically linked (with all default targets) (built locally)

Packages (13) mingw-w64-x86_64-clang-13.0.0-1 mingw-w64-x86_64-clang-analyzer-13.0.0-1 mingw-w64-x86_64-clang-tools-extra-13.0.0-1 mingw-w64-x86_64-compiler-rt-13.0.0-1 mingw-w64-x86_64-libc++-13.0.0-1 mingw-w64-x86_64-libc++abi-13.0.0-1 mingw-w64-x86_64-libunwind-13.0.0-1 mingw-w64-x86_64-lld-13.0.0-1 mingw-w64-x86_64-lldb-13.0.0-1 mingw-w64-x86_64-llvm-13.0.0-1 mingw-w64-x86_64-mlir-13.0.0-1 mingw-w64-x86_64-openmp-13.0.0-1 mingw-w64-x86_64-polly-13.0.0-1

Total Installed Size:  5061.76 MiB
Net Upgrade Size:      3916.98 MiB

CLANG32

Dynamically linked (with fewer targets)

Packages (13) mingw-w64-clang-i686-clang-13.0.0-1 mingw-w64-clang-i686-clang-analyzer-13.0.0-1 mingw-w64-clang-i686-clang-tools-extra-13.0.0-1 mingw-w64-clang-i686-compiler-rt-13.0.0-1 mingw-w64-clang-i686-libc++-13.0.0-1 mingw-w64-clang-i686-libc++abi-13.0.0-1 mingw-w64-clang-i686-libunwind-13.0.0-1 mingw-w64-clang-i686-lld-13.0.0-1 mingw-w64-clang-i686-lldb-13.0.0-1 mingw-w64-clang-i686-llvm-13.0.0-1 mingw-w64-clang-i686-mlir-13.0.0-1 mingw-w64-clang-i686-openmp-13.0.0-1 mingw-w64-clang-i686-polly-13.0.0-1

  Total Installed Size:  944.76 MiB
  Net Upgrade Size:      230.62 MiB

Statically linked (with all default targets)

Packages (13) mingw-w64-clang-i686-clang-13.0.0-1 mingw-w64-clang-i686-clang-analyzer-13.0.0-1 mingw-w64-clang-i686-clang-tools-extra-13.0.0-1 mingw-w64-clang-i686-compiler-rt-13.0.0-1 mingw-w64-clang-i686-libc++-13.0.0-1 mingw-w64-clang-i686-libc++abi-13.0.0-1 mingw-w64-clang-i686-libunwind-13.0.0-1 mingw-w64-clang-i686-lld-13.0.0-1 mingw-w64-clang-i686-lldb-13.0.0-1 mingw-w64-clang-i686-llvm-13.0.0-1 mingw-w64-clang-i686-mlir-13.0.0-1 mingw-w64-clang-i686-openmp-13.0.0-1 mingw-w64-clang-i686-polly-13.0.0-1

  Total Installed Size:  5466.04 MiB
  Net Upgrade Size:      4751.90 MiB

[jeremyd2019]

A third option that I thought of, but don't know how practical it would be, would be to split libLLVM into multiple DLLs (BUILD_SHARED_LIBS?), and then merge the import libraries into a libLLVM-13.dll.a so the consumer wouldn't have to know the difference.

[mati865]

BUILD_SHARED_LIBS conflicts with LLVM_LINK_LLVM_DYLIB and disables building of static libraries.
There is no such option right now so somebody would have to hack it together.

[jeremyd2019]

It looks like @mstorsjo only enables AArch64, arm, and X86 targets in his scripts: https://github.com/mstorsjo/llvm-mingw/blob/378525eb9b72a7deacdde084356926724238e231/build-llvm.sh#L235. Also looks like he uses DYLIB by default, unless explicitly asked not to.

[mati865]

Also looks like he uses DYLIB by default, unless explicitly asked not to.

Oh, thats a new thing in his LLVM 13 release.

[mstorsjo]

Yep, I took that into use recently, and the size benefits are massive.

If not using dylibs, you might be able to trim down some amount of the binary bloat by using minimal frontend wrappers, kinda like what I use. So instead of having two identical large binaries clang.exe and clang++.exe, you'd have a wrapper clang++.exe which invokes clang-13.exe --driver-mode=g++ which does the same as naming the binary that way. If proper symlinks were used throughout the whole build+packaging+unpacking process, that'd be a non-issue though.

[mstorsjo]

Oh, and also, there's some upstream effort lately to try to make the dylib build usable for MSVC too (by marking interfaces more correctly). Marking them properly (so it doesn't end up exporting needless internal symbols) probably would help a bit towards not hitting the 64k exports limit too.

[jeremyd2019]

I was able to expand the number of targets while still coming in under 65535 exports (CLANG32 came in at 64249). The list I came up with is AArch64;AMDGPU;ARM;AVR;BPF;Mips;MSP430;NVPTX;RISCV;WebAssembly;X86. It might be better to list what isn't present from the default list:

• Hexagon (a DSP in Qualcomm CPUs)
• Lanai (some Google-internal CPU)
• PowerPC (old Macs, currently mostly used in automotive applications according to Wikipedia)
• Sparc (old Sun, now Oracle)
• SystemZ (IBM mainframes)
• Xcore (🤷)

[jeremyd2019]

I merged with this target list. It just so happened a version of mesa with fixes for LLVM 13 was merged in the same window, so that was a good opportunity to avoid an extra rebuild of it.

[jeremyd2019]

As an update, I later had to disable MIPS too, as rebuilding llvm 13 with clang/llvm 13 resulted in even more exported symbols

[royqh1979]

Why add targets other than X86/WebAssembly?

Msys2 don't support other platforms, and it seems that no one will do cross-compile under msys2.

And it make programs that statically link with libclang very big (90M+), because clang_createIndex (in libclang) uses llvm::InitializeAllTargets()， and it depends all compiled llvm targets.

[royqh1979]

It seems that mingw-w64 llvm+clang built by qt only have X86 targets.

[MehdiChinoune]

[jeremyd2019]

I know that Zig expects all default targets to be enabled and fails to build if they are not. That ship has obviously already sailed. I don't know if anybody would be actively trying to cross-compile from msys2 or not, but I would try not to let my failure of imagination limit someone else's ingenuity 😉

[jeremyd2019]

I think I read in one of the LLVM docs that NVPTX and AMDGPU were not applicable for Windows. At least for me, I would rather have ARM than AMDGPU (but I am a luddite who only has very old hardware)

[jeremyd2019]

Hmm, I guess since the last time I checked it is possible to build mesa to use AMDGPU, but maybe not to actually run it on Windows? https://www.phoronix.com/scan.php?page=news_item&px=Mesa-Build-RADV-On-Windows

[pal1000]

This means I'll have to check if AMDGPU is available with MSYS2 the same way I do for MSVC and drop RADV from MinGW package if necessary.

I'll post an announcement on pal1000/mesa-dist-win and see if anyone complains.

[mati865]

If it works than we should keep it.

[MehdiChinoune]

Bad News!
With 15.0.0-rc2 I have to disable AMDGPU and WebAssembly on CLANG64 and BPF, MSP430 and RISCV on UCRT64.

[mstorsjo]

Just FYI, I've tried to take precautionary action on this issue (for my builds with only X86, ARM and AArch64, I'm also getting very close to the limit, so I would suspect that by LLVM 16, the dylib build would be broken for me too) - and am implementing something like hidden visibility for COFF; see https://reviews.llvm.org/D130120 and https://reviews.llvm.org/D130121. With those in place, we'd get down to around 35k exports in libLLVM.dll. But that'd only be for builds with clang.

If we land those on git main soon, I think it might be a bit late and disruptive for backporting into the LLVM 15 release at this point, but you might want to carry them as downstream patches already in 15.x.

[mati865]

Yeah, I've been considering backporting them since you published them to reviews.llvm.org 😉

[MehdiChinoune]

Just FYI, I've tried to take precautionary action on this issue (for my builds with only X86, ARM and AArch64, I'm also getting very close to the limit, so I would suspect that by LLVM 16, the dylib build would be broken for me too) - and am implementing something like hidden visibility for COFF; see https://reviews.llvm.org/D130120 and https://reviews.llvm.org/D130121. With those in place, we'd get down to around 35k exports in libLLVM.dll. But that'd only be for builds with clang.

If we land those on git main soon, I think it might be a bit late and disruptive for backporting into the LLVM 15 release at this point, but you might want to carry them as downstream patches already in 15.x.

How to use those patches, I have applied them but still exceeding symbols limit.
Should I bootstrap it (Build with just one target and then use that new compiler to build one with all targets)?
Should I pass --exclude-symbols to lld?

[mati865]

First you need to build LLVM 14 with all three patches (maybe also https://reviews.llvm.org/D130200 if you want to be still able to rebuild LLVM 14 with itself), then use that to build LLVM 15.
Alternatively you can build LLVM 15 with 3 patches applied and without shared libs, then use it to rebuild LLVM 15 with shared libs.

[mstorsjo]

Yep, exactly like this. And then, for the second stage, you need to apply another patch on llvm, to take the hidden visibility into use, that I haven’t posted for review yet. See https://martin.st/temp/0001-llvm-Use-hidden-visibility-when-building-on-MinGW.patch.

[MehdiChinoune]

New bad News!

1395/1397] Linking CXX shared library bin\libMLIR.dll
 FAILED: bin/libMLIR.dll tools/mlir-shlib/libMLIR.dll.a 
 cmd.exe /C "cd . && C:\msys64\ucrt64\bin\g++.exe -march=x86-64 -mtune=generic -O2 -pipe -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -Wimplicit-fallthrough -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wmisleading-indentation -ffunction-sections -fdata-sections -O3 -DNDEBUG  -pipe   -Wl,--gc-sections -shared -o bin\libMLIR.dll -Wl,--out-implib,tools\mlir-shlib\libMLIR.dll.a -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles\MLIR.rsp  && cd ."
 C:/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/12.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: error: export ordinal too large: 77799
 
 collect2.exe: error: ld returned 1 exit status
 ninja: build stopped: subcommand failed.

  [1388/1397] Linking CXX shared library bin\libMLIR.dll
  FAILED: bin/libMLIR.dll tools/mlir-shlib/libMLIR.dll.a 
  cmd.exe /C "cd . && C:\msys64\clang64\bin\clang++.exe -march=x86-64 -mtune=generic -O2 -pipe -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -ffunction-sections -fdata-sections -Werror=mismatched-tags -O3 -DNDEBUG  -pipe   -Wl,--gc-sections -shared -o bin\libMLIR.dll -Wl,--out-implib,tools\mlir-shlib\libMLIR.dll.a -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles\MLIR.rsp  && cd ."
  ld.lld: error: too many exported symbols (got 83452, max 65535)
  
  clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I don't know what to do with MLIR?

[mstorsjo]

Actually, a short import library with some static object files should have been supported - as long as those object files were named *.obj - but if not, you'd hit the same issue.

I finally got my head wrapped around this logic, and it seems possible to simplify it a lot, getting rid of the requirement of the static object files being named *.obj at the same time as fixing support for more than one DLL in the same import library. Can you testrun my patches for me (to see if they generally seem to work)? It's the two topmost commits at https://github.com/mstorsjo/binutils-gdb/commits/msvc-implib3.

[mati865]

It failed with:

ei386pe.c:153:12: error: static declaration of 'stricmp' follows non-static declaration
  153 | static int stricmp (const char *s1, const char *s2)
      |            ^~~~~~~
In file included from D:/msys64/mingw64/include/io.h:10,
                 from D:/msys64/mingw64/include/sys/stat.h:14,
                 from ../../binutils-2.39/ld/sysdep.h:35,
                 from ei386pe.c:35:
D:/msys64/mingw64/include/string.h:110:15: note: previous declaration of 'stricmp' with type 'int(const char *, const char *)'
  110 |   int __cdecl stricmp(const char *_Str1,const char *_Str2) __MINGW_ATTRIB_DEPRECATED_MSVC2005;
      |               ^~~~~~~

Then I renamed the functions and all calls within these files to __stricmp and it passed Rust tests with import libraries created with LLVM! This is wonderful.

I'm running full bootstrap with more tests to catch potential regressions so if I don't reply within an hour that will mean there were no regressions (from Rust POV at least).

[mstorsjo]

As you didn't follow up, I presume this worked well in the end! And I guess it works fine for other regular libraries too (gnu style import libraries and normal static libraries) as your build would have contained a bunch of them too. So in that case, I think I'll try to send this patch upstream (the first one about case insensitivity got approved already).

[mati865]

Yeah, 2 stage bootstrap + testsuite found no regressions.

And I guess it works fine for other regular libraries too (gnu style import libraries and normal static libraries) as your build would have contained a bunch of them too

Regular libraries were overwhelming majority in fact.

[mstorsjo]

FYI, this patch was approved and merged upstream now.