Move HDF4 to its own dispatch layer

[edhartnett]

In this PR the HDF4 code is moved to its own dispatch layer. This cleans a bunch of code out of the libsrc4 directory.

This is a very helpful pre-cursor to the user-defined binary format.

Fixes #748.
Part of #177.

(I also fixed the parallel build problems of tst_filters.sh, in order to get this past my CI. I can take that out if you need me to.)

[DennisHeimbigner]

There is an interesting question here. AFAIK, the hdf4 library does not
depend on the hdf5 library. This, to me implies a gap in our configuration.
It would be possible, I think, to enable the netcdf-4 data model without
enabling use of hdf5. Currently, the two are synonymous.

[edhartnett]

Yes that became clear to me as well. However, the current PR just continues the current state of affairs, which is that you can't get HDF4 without HDF5.

The user defined format feature I am going to introduce next also uses the netCDF-4 data model without using HDF5.

After that, a future modification of interest might be to separate the HDF5 from the netCDF-4 internal data model code in such a way that we could build in HDF4 and user-format support, without HDF5 being required.

The netCDF-4 internal data model is sufficiently general that many different user formats can make use of it. The advantage is that lots of functions (like all the inq functions) are just running off the internal data model - they will work for user-defined formats automatically, which is great. (As they also do for HDF4).

[WardF]

So maybe I missed it in the deluge of other information, but regarding a user-defined data format, is there an issue open with more information? We touched on it briefly before when you were here discussing PIO, remind me what the connection between the two is? Or is it a wholly separate feature we want to add to libnetcdf?

Additionally, why is it we can't get HDF4 without HDF5?

[edhartnett]

User-defined formats are described in #177. They will allow model groups which use proprietary formats to convert to netCDF, without breaking all their tools or losing access to data in the old format. Essentially they allow outside users to do what the HDF4 dispatch layer does for HDF4 - make it seem like netCDF.

The HDF4 read-only code has always been a part of the libsrc4 code, and always required --enable-netcdf-4 at configure time. It was actually written before the dispatch layer code was written. So to use HDF4 you also must have HDF5 installed.

After this PR is merged, with a little more hanky-panky with the code, it will be possible to allow HDF4 to be used without HDF5. I'm not sure if this will really be of much value in the case of HDF4. I don't know how many people are using it.

However, it would be very useful to do because it would be that user-defined formats could also be used without HDF5. I think this would be a common scenario. For example, a model group might want to convert from their internal format to netCDF CDF5. They should be able to do so without needing to install HDF5.

[WardF]

Thanks for helping organize my thoughts. I hadn't realized there was the dependency between hdf4/hdf5.

Thanks!

[DennisHeimbigner]

Separating the netcdf4 model from hdf5 is, IMO, pretty challenging, but useful.
I suspect that the right solution is to make the netcdf-4 model be the norm
which means that we need to eliminate all uses of USE_NETCDF4 and instead
start relying more on the error codes NC_ENOTNC4 and NC_ENOTBUILT.

[DennisHeimbigner]

I should note that I do something like this for dap4 already. Basically, I create
a hidden in-memory netcdf-4 file in which to store the netcdf4 translated
metadata for the dap4 model. Then I can redirect all of the metadata related
dispatch operations to that in-memory file. So I hypothesize that the key is
to separate out a netcdf4 metadata subsystem with some hooks into it
that allow for implementation specific operations to be used.

[edhartnett]

I don't think this will actually be that hard.

Basically the data structs (NC_VAR_INFO, etc.), and the functions that manipulate them, form an internal API that is used by different codes, like HDF4 and HDF5 code.

These functions are already pretty separate, with the HDF5 code in nc4hdf.c and most of the data layer code in ncfunc.c and nc4internal.c. So I think it will mostly be moving a bunch of what we call "internal" functions into a separate directory (libint) which will be the data layer code.

The remaining code in libsrc4 will be the HDF5 code. Then it should be pretty easy to ensure the build works with the internal model, and without HDF5.

I would like to get the currently working user-defined format code merged first, and the circle around and separate out the HDF5 code.

[DennisHeimbigner]

There may be an additional benefit to isolating the metadata code.
We have wanted to do lazy metadata access and have hypothesized that
in internal metadata module is necessary to do that.

[edhartnett]

I have actually been pondering how to do lazy metadata access.

I am thinking that if we read a group at a time, that would work...

But I have not seriously tried it yet. I agree that separating out the internal data code would be a helpful step, because isolating that code would be a nice simplification of the libsrc4 code. The less there is, the less chance of breaking something.

After I get the user-defined data code merged in, I will take a stab at separating out the data model internals.

[DennisHeimbigner]

I disagree about group at a time. Once you have the mechanism in place,
you can read any single item -- type,var,dim,attr,group -- at a time taking
dependencies into account. My speculation is that only doing lazy eval
of vars and attr is the big payoff.

[DennisHeimbigner]

Also, we have a specific case. A file with only the root group
and with 14mb of metadata. So reading per-group would buy
nothing for this file.

[edhartnett]

@DennisHeimbigner and other interested readers, I have created two new issues, #856 and #857, where the issues of separating the netCDF-4 internal data code from HDF5, and implementing lazy reads, can be discussed.

[edhartnett]

OK, something funny was going on with the cmake build, which was causing the failures.

The CMakeLists.txt file has this:

# Option to use HDF4
OPTION(ENABLE_HDF4 "Build netCDF-4 with HDF4 read capability(HDF4, HDF5 and Zlib required)." OFF)

So HDF4 should be off by default. But that was not the case. Not sure why, but after cmake is run, I see:

# Features
--------
NetCDF-2 API:		yes
HDF4 Support:		yes
NetCDF-4 API:		no

Not sure why this is happening.

So I added the else section below to expressly turn off HDF4 if netcdf-4 is not enabled:

# Build netCDF4
OPTION(ENABLE_NETCDF_4 "Enable netCDF-4" ON)
IF(ENABLE_NETCDF_4)
  SET(USE_NETCDF4 ON CACHE BOOL "")
  SET(ENABLE_NETCDF_4 ON CACHE BOOL "")
  SET(ENABLE_NETCDF4 ON CACHE BOOL "")
ELSE()
  SET(USE_HDF4_FILE_TESTS OFF)
  SET(USE_HDF4 OFF)  
  SET(ENABLE_HDF4_FILE_TESTS OFF)
  SET(ENABLE_HDF4 OFF)  
ENDIF()

@WardF if you know of a better way to deal with this, let me know.

[DennisHeimbigner]

Did you check to see if it gets turned off if the library is not available?
I do not recall seeing this problem before.

[edhartnett]

OK, the one travis test that failed did so due to a timeout. So I believe this PR is good to merge, @WardF, whenever you are ready.

[WardF]

It will probably be next week some time.

[edhartnett]

@WardF can we get this merged soon?

I would like to get the user-defined format work up for review. It will probably lead to some discussions, so I want to get it going.

Thanks,
Ed

[WardF]

There we go!

[edhartnett]

Thanks!