[MDAL] Add tools, drivers

[evetion]

Windows builds have less drivers (due to NetCDF missing) than other platforms: https://github.com/evetion/MDAL.jl/actions/runs/2934632776

[evetion]

It seems enabling the tools (mdalinfo executable) triggers linking errors on Linux (only):

[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_init'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_retval_tls'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_memmove'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_va_arg_tls'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_param_tls'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_warning_with_origin_noreturn'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_va_arg_overflow_size_tls'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_memset'
[20:38:22] /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root//usr/local/lib/libz.so.1: undefined reference to `__msan_memcpy'

[giordano]

Things are a bit in flux here trying to get memory sanitizer working in CI for Julia: JuliaLang/julia#46349

[evetion]

I'm not into sanitizers, is there a way to disable it for building the executables?

[giordano]

If there was a way to avoid mismatches we'd have put an automated fix in place already. The fact that there is no solution at the moment was the whole point of previous message. We're figuring out what to do as we go.

[Keno]

Just to close the circle, the version of BBB on Yggdrasil master was update to avoid the msan issue, so if this is rebased, it might be able to move forward.

[giordano]

This has been rerun already after #5391, and x86_64-linux-gnu is now passing. Now there are failures unrelated to msan in other platforms (well, they were there also before, since msan affected only x86_64-linux-gnu anyway)

[giordano]

The remaining error

[12:15:49] /opt/bin/i686-linux-gnu-libgfortran3-cxx03/i686-linux-gnu-g++ --sysroot=/opt/i686-linux-gnu/i686-linux-gnu/sys-root/  -Wall -Wno-long-long -pedantic -O3 -DNDEBUG CMakeFiles/mdalinfo.dir/mdalinfo.cpp.o -o mdalinfo  -Wl,-rpath,/workspace/srcdir/MDAL-release-0.9.4/mdal:/usr/local/lib: ../mdal/libmdal.so /opt/i686-linux-gnu/i686-linux-gnu/sys-root/usr/local/lib/libgdal.so /opt/i686-linux-gnu/i686-linux-gnu/sys-root/usr/local/lib/libxml2.so /opt/i686-linux-gnu/i686-linux-gnu/sys-root/usr/local/lib/libsqlite3.so 
[12:15:49] /opt/i686-linux-gnu/i686-linux-gnu/sys-root/usr/local/lib/libgdal.so: undefined reference to `__divmoddi4@GCC_7.0.0'
[12:15:49] collect2: error: ld returned 1 exit status

looks very much like JuliaPackaging/BinaryBuilderBase.jl#163, which is weird because it should have been fixed already.

[giordano]

#4979 increased the version of GCC used to build gdal from 6 to 7, so it may be a good idea to try and use gcc 7 here as well, but then I don't know how #5382 worked, if that's the issue

[evetion]

This adds building some Executables, those errors happen on linking those.

If I want to build this with NetCDF on Windows, should I also build NetCDF with expand_cxxstring_abis? Now I either get Dependency NetCDF_jll does not have a mapping for artifact NetCDF for platform x86_64-w64-mingw32-libgfortran3-cxx03, or without the expand_cxxstring_abis in this build, the Windows one does build with NetCDF, but complains about /agent/_work/1/s/M/MDAL/build/x86_64-w64-mingw32/Mx3j4jNY/x86_64-w64-mingw32-libgfortran3-cxx11/destdir/lib/libmdal.dll contains std::string values! This causes incompatibilities across the GCC 4/5 version boundary. To remedy this, you must build a tarball for both GCC 4 and GCC 5.

[evetion]

Odd, on x64-linux-musl, curl suddenly misbehaves with GCC7. now misbehaves with libexpat...
/usr/lib/libexpat.so.1: undefined reference to getrandom'`

[evetion]

And when cheating, and just checking whether a hdf5 library is available, we now also fail on all Windows platforms:

Could NOT find HDF5 (missing: HDF5_LIBRARIES) (found version "1.12.2")

Note sure if that's a bug, or there are explicit version checks or so.

[giordano]

now misbehaves with libexpat...
/usr/lib/libexpat.so.1: undefined reference to getrandom'

The problem is that something is pushing /usr/lib into the search path: https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=21839&view=logs&j=37ef6d6b-0cd0-533b-6dc1-e159d89496a0&t=138bfd31-39fd-5067-452b-ad7bf0cf1a7a&l=1269

[12:47:18] /opt/bin/x86_64-linux-musl-libgfortran4-cxx03/x86_64-linux-musl-g++ --sysroot=/opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/  -Wall -Wno-long-long -pedantic -O3 -DNDEBUG -lcurl CMakeFiles/mdal_translate.dir/mdal_translate.cpp.o -o mdal_translate  -Wl,-rpath,/workspace/srcdir/MDAL-release-0.9.4/mdal:/usr/local/lib: ../mdal/libmdal.so /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/lib/libgdal.so /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/lib/libxml2.so /opt/x86_64-linux-musl/x86_64-linux-musl/sys-root/usr/local/lib/libsqlite3.so 

Needless to say that's not a great idea. You can delete /usr/lib/libexpat.so* as a work around, but not having /usr/lib in the search path in the first place would be the best option.

Note sure if that's a bug, or there are explicit version checks or so.

Quoting from the documentation:

While below you will find some tips about common problems found when building packages in BinaryBuilder, keep in mind that if something fails during the build, there is not a magic recipe to fix it: you will need to understand what the problem is. Most of the time it's a matter of trial-and-error. The best recommendation is to access the build environment and carefully read the log files generated by the build systems: it is not uncommon that build systems would only print to screen misleading error messages, and the actual problem is completely different (e.g. "library XYZ can't be found", when the problem instead is that the command they run to look for library XYZ fails for unrelated reasons, for example for a wrong compiler flag used in the check). Having an understanding of what the build system is doing would also be extremely useful.

[evetion]

🎉

I'm still missing NetCDF on Windows though, which does work if we don't use expand_cxxstring_abis it seems.

[evetion]

This now adds NetCDF, HDF5 to x64 Windows and Linux, and to macOS, which roughly enables all drivers for the most popular platforms. I'm happy to merge this.