netCDF4 cannot read from little-endian HDF4 files

[timburgess]

Using OS X 10.9.5, the following example code will produce a little-endian HDF4 file which contains a simple incrementing 5x5 array. Specifically, binary ORing the defined datatype with DNFT_LITEND creates a little-endian file.

#include <stdio.h>
#include "mfhdf.h"

#define DIM1 5
#define DIM0 5
#define RANK 2
#define FILENAME "hdf_little-endian.hdf"
#define SDSNAME "data"

int main() {
  
    int32 sd_id, sds_id, istat, sd_index;
    int32 dims[2], start[2], edges[2], rank;
    int16 array_data[DIM0][DIM1];
    intn i, j, count;  
  
    start[0] = 0;
    start[1] = 0;
    edges[0] = DIM1;
    edges[1] = DIM0;
  
    // populate data array
    count = 0;
    for (j = 0; j < DIM0; j++) 
    {
        for (i = 0; i < DIM1; i++)
            array_data[j][i] = count++;
    }
  
    sd_id = SDstart(FILENAME, DFACC_CREATE);
    if (sd_id != FAIL)
        printf ("Opened %s\n", FILENAME);

    sds_id = SDcreate(sd_id, SDSNAME, DFNT_LITEND|DFNT_INT16, RANK, edges);
//  sds_id = SDcreate(sd_id, SDSNAME, DFNT_INT16, RANK, edges);

    istat = SDendaccess(sds_id);

    istat = SDend(sd_id);
    if (istat != FAIL)
        printf("closed\n");

       sd_id = SDstart(FILENAME, DFACC_WRITE);
    if (sd_id != FAIL)
        printf ("Opened %s\n", FILENAME);
  
    sd_index = 0;
    sds_id = SDselect(sd_id, sd_index);
  
    istat = SDwritedata(sds_id, start, NULL, edges, (VOIDP)array_data);
  
    istat = SDendaccess(sds_id);

    istat = SDend(sd_id);
    if (istat != FAIL)
        printf("closed\n");       
  
    return 0;
}

Using hdp dumpsds filename shows

File name: hdf_little-endian.hdf 

Variable Name = data
     Index = 0
     Type= little-endian format 16-bit signed integer
     Ref. = 2
     Compression method = NONE
     Rank = 2
     Number of attributes = 0
     Dim0: Name=fakeDim0
         Size = 5
         Scale Type = number-type not set
         Number of attributes = 0
     Dim1: Name=fakeDim1
         Size = 5
         Scale Type = number-type not set
         Number of attributes = 0
     Data : 
                0 1 2 3 4 
                5 6 7 8 9 
                10 11 12 13 14 
                15 16 17 18 19 
                20 21 22 23 24 

However ncdump fails to read the file with a NetCDF: Bad type ID.

This issue also impacts the python netCDF4 library as it fails with the same error, producing:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4.pyx", line 1503, in netCDF4.Dataset.__init__ (netCDF4.c:24382)
RuntimeError: NetCDF: Bad type ID

[WardF]

Hi, can you confirm if this issue still occurs?

[timburgess]

Still occurs. Just to be clear, I have built with the following steps from master

autoreconf -i
./configure --prefix=/usr/local --enable-hdf4
make
make check
make install

Using ncdump this produces:

ncdump hdf_little-endian.hdf
ncdump: hdf_little-endian.hdf: NetCDF: Bad type ID

I have put the file I am testing against at https://www.dropbox.com/s/utig3mtnjvhuiqx/hdf_little-endian.hdf?dl=0

[WardF]

Thank you for confirming; I had hoped that the fix for reading endianness from a netcdf4 file had cleared this up. I will investigate tomorrow and follow up here when I have some more information. Thanks!

Sent from my iPhone

On May 27, 2015, at 8:29 PM, Tim Burgess notifications@github.com wrote:

Still occurs. Just to be clear, I have built with the following steps from master

autoreconf -i
./configure --prefix=/usr/local --enable-hdf4
make
make check
make install
Using ncdump this produces:

ncdump hdf_little-endian.hdf
ncdump: hdf_little-endian.hdf: NetCDF: Bad type ID
I have put the file I am testing against at https://www.dropbox.com/s/utig3mtnjvhuiqx/hdf_little-endian.hdf?dl=0

—
Reply to this email directly or view it on GitHub.

[WardF]

Hi Tim, I've created an issue for this in the Unidata JIRA system; you can view it at https://bugtracking.unidata.ucar.edu/browse/NCF-332. I've narrowed down the issue and I think I have a fix in mind. I will post more information here when I have more to add, as well as updating the JIRA issue. I will also be creating a new branch, ncf332, where I will develop and test the fix.

Thanks again for the comprehensive bug report!

[timburgess]

Cheers Ward. Looking forward to getting this one nailed.

[WardF]

Ok, I have a tentative fix in branch ncf332, commit cf6d87f. I need to do some regression testing and documentation before merging into master, but barring any issues it looks like this will be fixed in the main development branch in short order, most likely tomorrow some time.

[WardF]

@timburgess Ok, the completed¹ fix is checked into branch ncf332, and will be merged into master later today. The test case you provided has been expanded and integrated into our test suite, thank you very much for that. I will close out this issue once I've merged the changes master. If you discover that it hasn't fixed the downstream issue in netCDF Python, let me know and I will reopen the issue.

1: "Completed" unless I find out there is still an error or other issue.

[WardF]

Fix merged into master in commit 27c97ba.

[timburgess]

Thanks Ward. Latest build of master works great.