netCDF4 cannot read from little-endian HDF4 files

[timburgess]

Using OS X 10.9.5, the following example code will produce a little-endian HDF4 file which contains a simple incrementing 5x5 array. Specifically, binary ORing the defined datatype with DNFT_LITEND create a little-endian file.

#include <stdio.h>
#include "mfhdf.h"

#define DIM1 5
#define DIM0 5
#define RANK 2
#define FILENAME "hdf_little-endian.hdf"
#define SDSNAME "data"

int main() {
  
    int32 sd_id, sds_id, istat, sd_index;
    int32 dims[2], start[2], edges[2], rank;
    int16 array_data[DIM0][DIM1];
    intn i, j, count;  
  
    start[0] = 0;
    start[1] = 0;
    edges[0] = DIM1;
    edges[1] = DIM0;
  
    // populate data array
    count = 0;
    for (j = 0; j < DIM0; j++) 
    {
        for (i = 0; i < DIM1; i++)
            array_data[j][i] = count++;
    }
  
    sd_id = SDstart(FILENAME, DFACC_CREATE);
    if (sd_id != FAIL)
        printf ("Opened %s\n", FILENAME);

    sds_id = SDcreate(sd_id, SDSNAME, DFNT_LITEND|DFNT_INT16, RANK, edges);
//  sds_id = SDcreate(sd_id, SDSNAME, DFNT_INT16, RANK, edges);

    istat = SDendaccess(sds_id);

    istat = SDend(sd_id);
    if (istat != FAIL)
        printf("closed\n");

       sd_id = SDstart(FILENAME, DFACC_WRITE);
    if (sd_id != FAIL)
        printf ("Opened %s\n", FILENAME);
  
    sd_index = 0;
    sds_id = SDselect(sd_id, sd_index);
  
    istat = SDwritedata(sds_id, start, NULL, edges, (VOIDP)array_data);
  
    istat = SDendaccess(sds_id);

    istat = SDend(sd_id);
    if (istat != FAIL)
        printf("closed\n");       
  
    return 0;
}

Using hdp dumpsds filename shows

File name: hdf_little-endian.hdf 

Variable Name = data
     Index = 0
     Type= little-endian format 16-bit signed integer
     Ref. = 2
     Compression method = NONE
     Rank = 2
     Number of attributes = 0
     Dim0: Name=fakeDim0
         Size = 5
         Scale Type = number-type not set
         Number of attributes = 0
     Dim1: Name=fakeDim1
         Size = 5
         Scale Type = number-type not set
         Number of attributes = 0
     Data : 
                0 1 2 3 4 
                5 6 7 8 9 
                10 11 12 13 14 
                15 16 17 18 19 
                20 21 22 23 24 

However ncdump fails to read the file with a NetCDF: Bad type ID. The python netCDF4 library also fails when using the Dataset constructor, producing:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4.pyx", line 1503, in netCDF4.Dataset.__init__ (netCDF4.c:24382)
RuntimeError: NetCDF: Bad type ID

I recognise that this is likely an upstream netCDF isuse but it would be good to identify which call is producing the error so the issue can be logged with the upstream providers.

[jswhit]

Tim: Could you create an issue for this at https://github.com/Unidata/netcdf-c/issues?

[timburgess]

Logged as Unidata/netcdf-c#113

[jswhit]

Could you try updating your netcdf C library from github master?

I'm wondering if the bug fix reported in Unidata/netcdf-c#112 helps.

[timburgess]

I've built from the netcdf-c github master on OS X using

autoreconf -vif
CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure --enable-hdf4 --enable-hdf4-file-tests
make
make check
make install

All tests passed.

However I still get:

ncdump hdf_little-endian.hdf 
ncdump: hdf_little-endian.hdf: NetCDF: Bad type ID
MBP:hdf4-endianess tim$ python
Python 2.7.9 (default, Jan  7 2015, 11:50:42) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from netCDF4 import Dataset
>>> ds = Dataset('hdf_little-endian.hdf')
Traceback (most recent call last):
  File "", line 1, in 
  File "netCDF4.pyx", line 1503, in netCDF4.Dataset.__init__ (netCDF4.c:24382)
RuntimeError: NetCDF: Bad type ID

For reference, the HDF4 library version I'm using is 4.2.10

[jswhit]

Oh well. Let's see if they respond to the ticket you created.

[WardF]

@jswhit The issue should hopefully be fixed now. More details over at Unidata/netcdf-c#113, or in our JIRA system at https://bugtracking.unidata.ucar.edu/browse/NCF-332, but long-story-short, the issue was identified and resolved. The fix has been merged into netcdf-c master. Unless somebody shows that the problem remains, this issue can probably be closed. Thanks again to @timburgess for the comprehensive bug report.

[jswhit]

Thanks @WardF!