No HDF4 support in provided NetCDF build

[Yujie-W]

netCDF 0.10.3 works well when reading a UInt16 NC file, but this error appears in 0.11.0

ERROR: NetCDF error code -128:
	NetCDF: Attempt to use feature that was not turned on when netCDF was built.
Stacktrace:
 [1] check at /home/user/.julia/packages/NetCDF/2DiO1/src/netcdf_helpers.jl:22 [inlined]
 [2] nc_open(::String, ::UInt16, ::Array{Int32,1}) at /home/user/.julia/packages/NetCDF/2DiO1/src/netcdf_c.jl:234
 [3] nc_open at /home/user/.julia/packages/NetCDF/2DiO1/src/netcdf_helpers.jl:103 [inlined]
 [4] open(::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool) at /home/user/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:911
 [5] open(::NetCDF.var"#47#48"{String}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/user/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:997
 [6] open at /home/user/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:997 [inlined]
 [7] #ncread#46 at /home/user/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:1050 [inlined]
 [8] ncread(::String, ::String) at /home/user/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:1050
 [9] top-level scope at REPL[4]:1

Here is the file for testing purpose: https://drive.google.com/file/d/19OEv2_R9DxO4I7K6MDK1e6yjxFocAXKQ/view?usp=sharing
The command I use, for example,

ncread("MOD15A2H.A2000049.h00v08.006.2015136143547.hdf", "Lai_500m")

[visr]

@Yujie-W was the issue resolved? I don't know what kind of feature this would be, isn't UInt16 (NC_USHORT) supported by default? https://www.unidata.ucar.edu/software/netcdf/docs/data_type.html

[Yujie-W]

@visr No, the problem is still there. I tried it on my pc and server. Here is the results
On my PC (manjaro) NetCDF v0.10

(@v1.5) pkg> st NetCDF
Status `~/.julia/environments/v1.5/Project.toml`
  [30363a11] NetCDF v0.10.3

julia> using NetCDF
julia> ncread("test.hdf", "Lai_500m");
ERROR: NetCDF error code -128:
	NetCDF: Attempt to use feature that was not turned on when netCDF was built.

On my pc (manjaro) NetCDF v0.11

(@v1.5) pkg> st NetCDF
Status `~/.julia/environments/v1.5/Project.toml`
  [30363a11] NetCDF v0.11.0

julia> using NetCDF
[ Info: Precompiling NetCDF [30363a11-5582-574a-97bb-aa9a979735b9]
julia> ncread("test.hdf", "Lai_500m");
ERROR: NetCDF error code -128:
	NetCDF: Attempt to use feature that was not turned on when netCDF was built.

On the server (redhat) NetCDF v0.10 Succeed

(@v1.5) pkg> st NetCDF
Status `~/.julia/environments/v1.5/Project.toml`
  [30363a11] NetCDF v0.10.3

julia> using NetCDF
julia> ncread("test.hdf", "Lai_500m");

On the server (redhat) NetCDF v0.11

(@v1.5) pkg> st NetCDF
Status `~/.julia/environments/v1.5/Project.toml`
  [30363a11] NetCDF v0.11.0

julia> using NetCDF
[ Info: Precompiling NetCDF [30363a11-5582-574a-97bb-aa9a979735b9]
julia> ncread("test.hdf", "Lai_500m");
ERROR: NetCDF error code -128:
	NetCDF: Attempt to use feature that was not turned on when netCDF was built.

[Yujie-W]

Think it is due to the nctcdf version. I have netcdf v4.7.4 on my PC, here is the output

~/Downloads >>> ncdump -v
netcdf library version 4.7.4 of Apr 10 2020 23:29:06 $

~/Downloads >>> ncdump -h test.hdf
ncdump: test.hdf: test.hdf: NetCDF: Attempt to use feature that was not turned on when netCDF was built.

The server has an old version of netcdf

$ ncdump -v
netcdf library version 4.3.3.1 of Dec 10 2015 16:44:18 $
$ ncdump -h test.hdf
netcdf test {
dimensions:
	YDim\:MOD_Grid_MOD15A2H = 2400 ;
	XDim\:MOD_Grid_MOD15A2H = 2400 ;
variables:
	ubyte Fpar_500m(YDim\:MOD_Grid_MOD15A2H, XDim\:MOD_Grid_MOD15A2H) ;
... ...

Now sure what is going on. My guess is that NetCDF v0.10 uses linux system netcdf, so that it succeeds on the server but fails on my PC. NetCDF v0.11 uses NetCDF_jll (build on netcdf v4.7.4), so that it fails on both my PC and server.

[WardF]

The .hdf suffix makes it difficult to guess what the underlying file type is; is it hdf4, hdf5? Or is it a netCDF binary file with the .hdf extension? On the system that works for you, what is the output from ncdump -k test.hdf ?

[Yujie-W]

@WardF

$ ncdump -k test.hdf
netCDF-4

[WardF]

I have little familiarity with julia, but I notice the output [ Info: Precompiling NetCDF [30363a11-5582-574a-97bb-aa9a979735b9]. Assuming this is configuring and building the binary libnetcdf, there should be a file libnetcdf.settings generated somewhere. If so, are you able to locate and perhaps share the file? That will provide some information regarding how the library was compiled and with what options enabled.

[WardF]

All right, I was able to duplicate this using the .hdf file you linked to in the esupport system over at Unidata. The issue is that this is a .hdf4 file and, if libnetcdf is built without enabling hdf4 read-only support, the ncdump utility doesn't know how to read the file. So it seems like the issue is to do with how libnetcdf is being precompiled. I'm not certain how to ensure that hdf4 support would be enabled, but this should at least be a step in the right direction.

[meggart]

In the past we were using either system NetCDF binaries or used conda binaries as fallback and it looks like on your system this included support for hdf4 files.

However, with NetCDF 0.11 we switched to a custom NetCDF build using Yggdrasil and the Artifact system, here a link to the file that contains most of the current build script: https://github.com/JuliaPackaging/Yggdrasil/blob/master/N/NetCDF/common.jl and hdf4 is not supported. If there is a strong need to support HDF4, this would be the place to add it, but first we would need an hdf4 artifact, which is probably a nightmare to cross-compile. Related issues and PRs are:

#123
JuliaPackaging/Yggdrasil#2402
JuliaPackaging/Yggdrasil#1090

[visr]

Assuming this is configuring and building the binary libnetcdf, there should be a file libnetcdf.settings generated somewhere.

The binaries are downloaded from here: https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl/releases/tag/NetCDF-v400.702.400%2B0

This includes a libnetcdf.settings with these contents:

# NetCDF C Configuration Summary
==============================

# General
-------
NetCDF Version:		4.7.4
Dispatch Version:       2
Configured On:		
Host System:		aarch64-unknown-linux-gnu
Build Directory: 	/workspace/srcdir/netcdf-c-4.7.4
Install Prefix:         /workspace/destdir

# Compiling Options
-----------------
C Compiler:		/opt/bin/aarch64-linux-gnu-libgfortran3-cxx03/cc
CFLAGS:			
CPPFLAGS:		-I/workspace/destdir/include
LDFLAGS:		-L/workspace/destdir/lib
AM_CFLAGS:		
AM_CPPFLAGS:		
AM_LDFLAGS:		
Shared Library:		yes
Static Library:		no
Extra libraries:	-lhdf5_hl -lhdf5 -lm -ldl -lz -lcurl 

# Features
--------
NetCDF-2 API:		yes
HDF4 Support:		no
HDF5 Support:		yes
NetCDF-4 API:		yes
NC-4 Parallel Support:	no
PnetCDF Support:	no
DAP2 Support:		yes
DAP4 Support:		yes
Byte-Range Support:	no
Diskless Support:	yes
MMap Support:		no
JNA Support:		no
CDF5 Support:		yes
ERANGE Fill Support:	no
Relaxed Boundary Check:	yes
SZIP Support:           no
SZIP Write Support:     no
Parallel Filters:       yes 

So indeed, as you have already found out, HDF4 is not supported in these builds at the moment.

[Yujie-W]

I kind of remember I opened an issue about HDF4 support somewhere, and here it is. I opened another one at NCDatasets.jl as well, tried a sway around solution, see Alexander-Barth/NCDatasets.jl#217.

Maybe you can add the fallback solution back, e.g.,

import NetCDF_jll

const LIBNETCDF  = deepcopy(NetCDF_jll.libnetcdf);

function switch_netcdf_lib!(; use_default::Bool = true, user_defined::String = "$(homedir())/.julia/conda/3/lib/libnetcdf.so")
    if use_default
        NetCDF_jll.libnetcdf = LIBNETCDF;
    else
        if isfile(user_defined)
            NetCDF_jll.libnetcdf = user_defined;
        else
            @warn "File '$(user_defined)' not found!";
            @info "Hint: You may libnetcdf shipped with Conda.jl using Conda.add(\"libnetcdf\"). A version above 4.8.1 is recommended.";
            @warn "The file '$(user_defined)' does not exist, please make sure you have provided the correct path!";
        end;
    end;

    return nothing
end