HDF5: Cross-compile

[eschnett]

Build HDF5 by cross-compiling instead of by downloading Conda packages.

[eschnett]

Now that this is working I'd be happy about feedback:

Questions:

• Which HDF5 API should I use? (Does this matter?)
• The current HDF5 packages depend on OpenSSL and libCURL. This one doesn't. Why? What am I missing?
• How should I test this change?

[giordano]

CC @mkitti

[mkitti]

Oh neat. We're still filling in predetermined values though, right?

[mkitti]

Now that this is working I'd be happy about feedback:

Questions:

• Which HDF5 API should I use? (Does this matter?)

You should use the C API.

• The current HDF5 packages depend on OpenSSL and libCURL. This one doesn't. Why? What am I missing?

This is due to the read-only S3 driver, ROS3. If you enable that you will pick up the dependency.

Also, we should enable SZIP.

• How should I test this change?

[mkitti]

Cc: @simonbyrne @musm

[mkitti]

@simonbyrne are you interested in doing anything with MPI here?

[eschnett]

@mkitti Yes, we're using predetermined values. The values mostly describe floating point layouts, in particular for long double.

For Linux I used Docker and QEMU to determine the values, for Apple they are documented, and for Windows they seem to be the same as for Linux.

HDF5 offers different APIs (called API_16, API_18, API_110 etc.). Does it matter which we choose? I think this API is mostly handled via #define in the header files.

I can add the S3 driver and SZIP.

I also want to enable the C++ and Fortran APIs so that other packages can be built against this HDF5 library. I'm also thinking of enabling MPI.

[simonbyrne]

Woohoo! Thanks @eschnett!

The values mostly describe floating point layouts, in particular for long double.

For Linux I used Docker and QEMU to determine the values, for Apple they are documented, and for Windows they seem to be the same as for Linux.

I don't think that's right: long double on Windows is 64-bit (https://learn.microsoft.com/en-us/previous-versions/9cx8xs15(v=vs.140)?redirectedfrom=MSDN), whereas it is 80-bit on Linux x86_64 (amd64).

HDF5 offers different APIs (called API_16, API_18, API_110 etc.). Does it matter which we choose? I think this API is mostly handled via #define in the header files.

These are for backward compatibility (i.e. API_16 is the HDF5 1.6 API). I don't quite understand how they work (https://docs.hdfgroup.org/hdf5/v1_12/api-compat-macros.html), but we should probably just use the most recent API.

I'm also thinking of enabling MPI.

That would be very handy!

[simonbyrne]

I don't think that's right: long double on Windows is 64-bit (https://learn.microsoft.com/en-us/previous-versions/9cx8xs15(v=vs.140)?redirectedfrom=MSDN), whereas it is 80-bit on Linux x86_64 (amd64).

Ah, that is for MSVC; we're using mingw so i guess it is 80-bit (this is all confusing)

[Alexander-Barth]

Looks like NetCDF 4.9.2 supports HDF5 1.14: https://docs.unidata.ucar.edu/netcdf-c/current/RELEASE_NOTES.html

I just upgraded NetCDF_jll to version 4.9.2 (but leaving the HDF5 compat bound unchanged Dependency(PackageSpec(name="HDF5_jll"), compat="1.12.2").

#6702

I did not notice any issues running my package NCDatasets.jl with this NetCDF_jll.

https://github.com/Alexander-Barth/NCDatasets.jl/actions/runs/4830631528

There is an issue with NetCDF_jll and HDF5 1.12.1 I believe.

Yes, I think there was an issue on Windows if I remember correctly.

[giordano]

I did not notice any issues running my package NCDatasets.jl with this NetCDF_jll.

Can you try with https://github.com/eschnett/HDF5_jll.jl?

[giordano]

But the fact that the soname changed
https://github.com/JuliaBinaryWrappers/HDF5_jll.jl/blob/450424f8e1fb2fd7f77a0d60a39674fe8621ea35/src/wrappers/x86_64-linux-gnu.jl#L8
vs
https://github.com/eschnett/HDF5_jll.jl/blob/03fdb88842cacf5d2502a766e79cdf3d08652f90/src/wrappers/x86_64-linux-gnu-libgfortran5-cxx11-mpi%2Bmpich.jl#L11
(why this even changed?!?!?) pretty much means that they are incompatible and we have to set compat bounds everywhere.

[eschnett]

So I did this #6707, with a skip build. How do I do the other part, the "retro cap the registry"?

[mkitti]

So I did this #6707, with a skip build. How do I do the other part, the "retro cap the registry"?

See what I did here for OpenSSL:
JuliaRegistries/General#78542

@DilumAluthge also has this tool: https://github.com/JuliaRegistries/RetroCap.jl

[mkitti]

HDF5 1.14.1 is now out.

[eschnett]

Of course it is... I'm busy with another project this week so there won't be progress, and the next steps are adding the compat bounds anyway.

[mkitti]

I would not recommend shifting over to HDF5 1.14.1 in this pull request.

[eschnett]

I agree.

[mkitti]

General registry cap at JuliaRegistries/General#84160

[giordano]

So now we should be good to go?

[mkitti]

As a HDF5.jl maintainer, this is good to go.

[musm]

@eschnett I wanted to bring to attention the latest changes in HDF5 (e.g., HDFGroup/hdf5#3104)
which should help simplify the HDF5 cross-compilation support.

[mkitti]

Oh we're quite aware of that pull request. It may even be a direct consequence of this pull request since I told them what we were doing a few weeks before they committed to it.