Superfluous HDF5 dependency

[nschloe]

Commit 5fa56b4 introduces a dependency on HDF5 which might not actually by necessary. Note that HDF5 symbols are already pulled in by the dependency on netCDF.
Do you use HDF5 directly and explicitly in netCDF-Fortran?

[WardF]

If the NetCDF library is static we require a method to specifically link against the downstream hdf5 library dependency (assuming it is required). This is still a work in progress and will be resolved before our next release.

Sent from my iPhone

On Jul 25, 2014, at 3:24 AM, Nico Schlömer notifications@github.com wrote:

Commit 5fa56b4 introduces a dependency on HDF5 which might not actually by necessary. Note that HDF5 symbols are already pulled in by the dependency on netCDF.
Do you use HDF5 directly and explicitly in netCDF-Fortran?

—
Reply to this email directly or view it on GitHub.

[WardF]

We cannot assume that NetCDF-c was built with cmake unfortunately or that cmake information is available; the majority of users still use automake.

Sent from my iPhone

On Jul 25, 2014, at 3:24 AM, Nico Schlömer notifications@github.com wrote:

Commit 5fa56b4 introduces a dependency on HDF5 which might not actually by necessary. Note that HDF5 symbols are already pulled in by the dependency on netCDF.
Do you use HDF5 directly and explicitly in netCDF-Fortran?

—
Reply to this email directly or view it on GitHub.

[nschloe]

I just skimmed through netCDF-Fortran and found no reference to HDF5 at all. I just compiled it without HDF5 and it all links cleanly, no missing symbols. The HDF5 dependency really looks superfluous.

What was your motivation for linking against it in the first place?

[WardF]

Adding a quick link to the relevant issue, #17.

Assumption

The following assumes that libnetcdf has been built statically, with downstream dependencies on hdf5 and hdf5_hl.

We cannot assume that libnetcdf was built using CMake.

Automake-based builds.

When building netcdf-fortran and linking against a static libnetcdf, the tests for various nc_ functions fail, because the test programs fail to link. They fail to link because of undefined references in the static netcdf library, present in libhdf5 and/or libhdf5_hl. The failure of these tests leads to a variable going undefined, which results in a number of other issues.

If the variable is defined by hand, we then see a number of linker errors due to the undefined references in libnetcdf.a.

This issue is fixed by assigning LIBS="-lhdf5 -lhdf5_hl" at configure time, e.g.

$ LIBS="-lhdf5 -lhdf5_hl" ./configure

Other downstream dependencies, such as libz and libm are inherited from the hdf5 libraries, because they have been built as shared libraries and contain information relating to their dependencies.

CMake-based builds.

When building with CMake, we will not see the same issue when testing libnetcdf for nc_open, etc, because we don't make these checks. We probably should, but that will come later. When trying to link libnetcdff.dylib, however, we see the following:

Undefined symbols for architecture x86_64:
  "_H5Aclose", referenced from:
  _nc4_rec_read_metadata in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _nc4_rec_read_metadata_cb in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _nc4_rec_write_metadata in libnetcdf.a(libnetcdf4_la-nc4hdf.o)
  _write_attlist in libnetcdf.a(libnetcdf4_la-nc4hdf.o)
  _nc4_rec_write_groups_types in libnetcdf.a(libnetcdf4_la-nc4hdf.o)
  _write_netcdf4_dimid in libnetcdf.a(libnetcdf4_la-nc4hdf.o)

...

  "_H5open", referenced from:
  _nc4_reopen_dataset in libnetcdf.a(libnetcdf4_la-nc4var.o)
  _NC4_create in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _NC4_open in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _nc4_rec_read_metadata_cb in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _get_netcdf_type in libnetcdf.a(libnetcdf4_la-nc4file.o)
  _nc4_get_hdf_typeid in libnetcdf.a(libnetcdf4_la-nc4hdf.o)
  _nc4_put_vara in libnetcdf.a(libnetcdf4_la-nc4hdf.o)
  ...
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [fortran/libnetcdff.6.0.1.dylib] Error 1
make[1]: *** [fortran/CMakeFiles/netcdff.dir/all] Error 2

This is to be expected since libnetcdf.a does not carry information regarding dependencies with it, and so the linker is not linking against the inherited dependencies. It is trivial to confirm that these symbols are undefined in the library using the nm utility on OSX and/or linux.

Unfortunately, CMake does not have an equivalent to automake's LIBS (that I can find). A better solution may be to implement a LIBS-like option for cmake-based builds, but I have not decided yet whether to go this route or not.

Other thoughts

When using automake, libtool is supposed to interpret the installed .la file to infer dependencies, even for a static library. That is not happening on either my OSX or linux system. I don't know if it is a libtool issue, an automake issue, or how we are configuring/installing netcdf-c. Even if we get it fixed, we cannot assume that there will be a .la file generated for libnetcdf, or that a user will be using automake to build libnetcdff.

It is likely that I will reverse the work I've done automatically looking for hdf5 dependencies and instead implement a LIB-like variable. If you know of such a flag I can use instead, please let me know, thanks in advance!

I'm going to close out this issue because when dealing with static libraries that don't contain information about inherited dependencies, there needs to be a way to tell the linker what objects define the relevant functions.

[nschloe]

I understand the problem now.

The issue lies in the fact that it is not clear how netCDF-C is supposed to transmit the information "I'm using symbols from -lhdf5 and -lhdf5_hl" to consumers, e.g., netCDF-Fortran's test executables.

In a CMake world, this information is supposed to travel on through the export files. Is netCDF-C using those? The man page http://www.cmake.org/cmake/help/v3.0/module/CMakePackageConfigHelpers.html can help.

There must be something similar for automake as well. How are export configurations created there?

[nschloe]

Not sure why this is closed. IMHO, we can close this if Unidata/netcdf-c#74 gets merged, we use this in the CMake build here and finally remove the dependency.

[WardF]

It was closed as a duplicate of #17, which remains open for the moment. I'll take a look at Unidata/netcdf-c#74 and merge it shortly, but it only solves the issue if a user is using cmake; sadly, the majority of our users are still using autotools, so we can't assume cmake is being used to build netcdf-c. Hence the addition of the 'NC_EXTRA_DEPS' for use when cmake is not used to build netcdf-c.

[nschloe]

but it only solves the issue if a user is using cmake

That's right. Since you are supporting both automake and CMake, and that on both netCDF and netCDF-Fortran, basically we need to have both CMake and Makefile exports for both CMake and Makefile builds. I guess I could fix up a Makefile export from a netCDF CMake build, too, but that won't be relocatable (which might not be a big deal).

[nschloe]

Another possibility that just came to my mind is that in netCDF-Fortran's CMakeLists.txt, you could FIND_PACKAGE(netCDF), and if it was found, use the EXTRA_LIBS information present there. Otherwise, add the HDF5 dependency.