-
Notifications
You must be signed in to change notification settings - Fork 558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add GDAL builder #293
add GDAL builder #293
Conversation
This gets us past a failure during the `configure` step where it checks the PROJ dependency by compiling test programs. This led to this failure related to `libstdc++`: ``` configure:24103: checking for proj_create_from_wkt in -lproj configure:24128: c++ -std=c++11 -o conftest -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 conftest.cpp -lproj -L/workspace/destdir/lib -lproj -lz -L/workspace/destdir -L/workspace/destdir/lib -lpthread -lm -lrt -ldl >&5 /workspace/destdir/lib/libproj.so: undefined reference to `__cxa_throw_bad_array_new_length@CXXABI_1.3.8' /workspace/destdir/lib/libproj.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> ```
Sweet! Looking forward to this. |
To make export LDFLAGS="${libdir}" (this is fine, it's common to have to do this), but also do cp ${libdir}/libproj_6_2.dll ${libdir}/libproj.dll otherwise |
Yeah sorry, this one was not yet included from GDALBuilder. Cannot find it anymore, but I believe that I saw this same trick being done in the GDAL repository. MinGW builds should work, there are Travis configurations set here: https://github.com/OSGeo/gdal/tree/v3.0.2/gdal/ci/travis/mingw_w64 |
Ok after fed5e11 we actually get 20 passing builds, 6 failing builds, overview at https://github.com/JuliaPackaging/Yggdrasil/pull/293/checks?check_run_id=349623402 I went through the 6 failing ones, and there are two different issues:
|
The problem with x86_64-linux-musl might be that it isn't doing a real cross-compilation and making a mess with different C++ libraries. From the log:
Edit: well, there is only a single |
Co-Authored-By: Mosè Giordano <giordano@users.noreply.github.com>
Ok, after a8937ca the Windows builds are still failing, but now at least similarly to the |
G/GDAL/build_tarballs.jl
Outdated
|
||
if [[ ${target} == *mingw* ]]; then | ||
export LDFLAGS="-L${libdir}" | ||
cp ${libdir}/libproj_6_2.dll ${libdir}/libproj.dll |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The more I think about this the more I'm convinced that we shouldn't do this here: in any case ${libdir}/libproj.dll
must not be in the tarball generated by this builder, but if we keep this line and remove the file at the end of the builder, this file will be a missing dependency for whatever needs it. I think we should either copy the file in PROJ_jll
or do something to convince GDAL to link against the right file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
libgdal
needs libproj
. However, due to PROJ_jll
with it's libproj_6_2.dll
being a dependency of GDAL_jll
, won't libproj_6_2.dll
already be dlopen
ed before libgdal
and thus everything is fine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's not a matter of being dlopened, when the linker is passed the option -lfoo
it'll look for a file called libfoo.${dlext}
(I think on Windows the "lib" is not necessary), with some variations to allow for the version number. Here the file is called libproj_6_2.dll
, since the Makefile is compiling with -lproj
the linker can't find libproj.dll
(probably it would allow for libproj-62.dll
). I think the dynamic loader follows similar rules as the linker: if the library is linked to libproj.dll
, libproj_6_2.dll
won't be an acceptable replacement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. But in GDALBuilder we used the same trick and dit not distribute libproj.dll
, only libproj_6_1.dll
. And it does find it:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I can see you're not deleting the file at the end, so it's probably in GDAL tarball, which is definitely something that we don't want. As I said before, the linker should allow for the version of the package to be in the file name, but I've never seen it as libproj_6_1
, I'm surprised the loader can find it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm in the old GDAL/lib
there is libproj.dll.a
, but that's all, and if I move that file away it still finds it just fine.
So at least we should add a rm ${libdir}/libproj.dll
after compilation in this build script to prevent getting that file out of this build? And if we get issues we try to fix the name in the PROJ build script?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The libraries for Windows are under bin, not lib
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I know, I just meant that under bin there is only libproj_6_1.dll
, nothing else. So I gues the loader on Windows is sufficiently flexible, just not during the build, which is why we needed to rename it before the build only.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have a look at b7eae71, if you don't like we can hard-reset it. However, I think this is a better way to deal with this, instead of copying files around.
In an ideal world, PROJ would provide a pkgconfig file and we would get from there the name of the library, instead of doing this mess.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Have you considered submitting this patch upstream to https://github.com/OSGeo/gdal/commits/master/gdal/configure.ac?
I think the remaining problem is that # Libraries that this one depends upon.
dependency_libs=' /workspace/destdir/lib/libgeos.la /workspace/destdir/x86_64-w64-mingw32/lib/../lib/libstdc++.la' Deleting all libtool libraries with (like in Gnome_themes_extra) # Clear out `.la` files since they're often wrong and screw us up
rm -f ${prefix}/lib/*.la lets me build successfully GDAL, but I get a lot of libtool error messages like
which may however be harmless for us |
I think they are. I always ignore libtool whenever I can. ;) |
Great, it's looking much greener on the Azure side. For Windows and OSX I see some like this, but I assume it's harmless:
The only failure is
To me it looks like |
Agreed.
I think this is an accident having to do with linking differences between platforms and ABIs. I'm pretty sure we would have runtime problems on the other platforms, but for some reason the win32 platform is throwing a compile-time problem (perhaps because GCC 4 on win32 actually cannot use cxx11-style strings, whereas other platforms do have the symbols in their compiler support libraries) |
* [GEOS] expand_cxxstring_abis Ref #293 (comment) * expand supported platforms directly
Still same error after rebuilding GEOS with expanded C++ ABIs 🤔 |
Odd. The reason it is still the same error is that the https://github.com/JuliaBinaryWrappers/GEOS_jll.jl/releases/tag/GEOS-v3.8.0%2B2 |
Are the libraries identical bit-by-bit? Did we already achieve reproducible builds? |
Haha yeah, bit by bit. Didn't test every platform, only GEOS.v3.8.0.aarch64-linux-musl and |
Interestingly enough, I cannot generate reproducible builds locally: two different builds for the same target platforms generate non-identical libraries. Also the libraries for the two cxx* variants are different. |
This seems like it is quite close. What do we need to do to get it in? |
I think we need to get a cxx03 build of GEOS first, which hangs on #316. |
By setting For Windows the problem was that PROJ couldn't be found, but we minor version was bumped in #394, hopefully that's the only issue. For All in all, we may be very close to the end! |
Amazingly enough,
|
As far as I can see, this is the only platform showing the infamous warning. It looks like when the target is Edit: continue to JuliaPackaging/BinaryBuilder.jl#604 (comment) |
Includes a few build bug fixes: https://github.com/OSGeo/gdal/blob/v3.0.3/gdal/NEWS
Awesome, finally a different error not related to C++11 string ABI 😂
Probably we need to build with gcc 6, geos has been built with gcc 6 |
An attempt to build GDAL with a rather minimal set of dependencies. We'd like to add more, but only after this builds well, taking it one step at a time.
Going over it with @giordano on Slack we already managed to bypass
configure
errors (PROJ dependency checks) by addingCompilerSupportLibraries_jll
, see the second commit. Unfortunately it does not seem possible to bypass these checks withconfigure
arguments.Right now, on
x86_64-linux-gnu
, this hits:Which is odd since there is no
/workspace/destdir/x86_64-linux-gnu/
directory. Not sure where that path comes from or how to fix it. I tried to add--without-libtool
toconfigure
, and with that I got a fully succesfull build onx86_64-linux-gnu
. However it did not help on for instancei686-linux-musl
, so I left it out here. There I got this error:For that I still tried
; preferred_gcc_version=v"6"
(like GEOS_jll), but that didn't change anything.Relevant links
https://github.com/JuliaGeo/GDALBuilder
https://github.com/JuliaGeo/GDAL.jl
https://github.com/OSGeo/gdal/
https://gdal.org/