Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: NetCDF: Filter error: undefined filter encountered #2636

Open
tayloj25 opened this issue Feb 23, 2023 · 8 comments
Open

Error: NetCDF: Filter error: undefined filter encountered #2636

tayloj25 opened this issue Feb 23, 2023 · 8 comments

Comments

@tayloj25
Copy link

I am using netcdf-c-4.9.1 with hdf5-1.12.2 and zlib-1.2.13. I am attempting to run quite a similar example to the pres_temp_4D_rd.c example on the unidata website. My dimensions are slightly different though. My code appears as below:

#include <stdio.h>
#include <string.h>
#include <netcdf.h>

/* This is the name of the data file we will read. */
#define FILE_NAME "pressure_data.nc"

#define NDIMS 4
#define NLAT 45
#define NLON 57
#define LAT_NAME "y"
#define LON_NAME "x"
#define NREC 1
#define REC_NAME "time"
#define LVL_NAME "z"
#define NLVL 1

/* Names of things. */
#define PRES_NAME "pressure"
#define UNITS "units"
#define DEGREES_EAST "degrees_east"
#define DEGREES_NORTH "degrees_north"

/* These are used to calculate the values we expect to find. */
#define SAMPLE_PRESSURE 101403.1
#define START_LAT 26.0
#define START_LON -84.0

/* For the units attributes. */
#define UNITS "units"
#define PRES_UNITS "Pa"
#define LAT_UNITS "degrees_north"
#define LON_UNITS "degrees_east"
#define MAX_ATT_LEN 80

/* Handle errors by printing an error message and exiting with a

  • non-zero status. */
    #define ERR(e) {printf("Error: %s\n", nc_strerror(e)); return 2;}

int
main()
{
int ncid, pres_varid;
int lat_varid, lon_varid;

/* The start and count arrays will tell the netCDF library where to
read our data. */
size_t start[NDIMS], count[NDIMS];

/* Program variables to hold the data we will read. We will only
need enough space to hold one timestep of data; one record. */
float pres_in[NLVL][NLAT][NLON];
// float temp_in[NLVL][NLAT][NLON];

/* These program variables hold the latitudes and longitudes. */
float lats[NLAT], lons[NLON];

/* Loop indexes. */
int lvl, lat, lon, rec, i = 0;

/* Error handling. */
int retval;

/* Open the file. */
if ((retval = nc_open(FILE_NAME, NC_NOWRITE, &ncid)))
ERR(retval);

/* Get the varids of the latitude and longitude coordinate
* variables. */
if ((retval = nc_inq_varid(ncid, LAT_NAME, &lat_varid)))
ERR(retval);
if ((retval = nc_inq_varid(ncid, LON_NAME, &lon_varid)))
ERR(retval);

/* Read the coordinate variable data. */
if ((retval = nc_get_var_float(ncid, lat_varid, &lats[0])))
ERR(retval);
if ((retval = nc_get_var_float(ncid, lon_varid, &lons[0])))
ERR(retval);

/* Check the coordinate variable data. */
for (lat = 0; lat < NLAT; lat++)
if (lats[lat] != START_LAT + 5.*lat)
return 2;
for (lon = 0; lon < NLON; lon++)
if (lons[lon] != START_LON + 5.*lon)
return 2;

/* Get the varids of the pressure and temperature netCDF
* variables. */
if ((retval = nc_inq_varid(ncid, PRES_NAME, &pres_varid)))
ERR(retval);

/* Read the data. Since we know the contents of the file we know
* that the data arrays in this program are the correct size to
* hold one timestep. */
count[0] = 1;
count[1] = NLVL;
count[2] = NLAT;
count[3] = NLON;
start[1] = 0;
start[2] = 0;
start[3] = 0;

/* Read and check one record at a time. */
for (rec = 0; rec < NREC; rec++)
{
start[0] = rec;
if ((retval = nc_get_vara_float(ncid, pres_varid, start,
count, &pres_in[0][0][0])))
ERR(retval);

/* Check the data. */
   i = 0;
   for (lvl = 0; lvl < NLVL; lvl++)
                                            for (lat = 0; lat < NLAT; lat++)
                                                            for (lon = 0; lon < NLON; lon++)
                                                            {
                                                                            if (pres_in[lvl][lat][lon] != SAMPLE_PRESSURE + i)
                                                                                            return 2;
                                                                            //if (temp_in[lvl][lat][lon] != SAMPLE_TEMP + i)
                                                                            //            return 2;
                                                                            i++;
                                                            }

} /* next record */

/* Close the file. */
if ((retval = nc_close(ncid)))
                            ERR(retval);

return 0;
}

I am getting an error indicating that an undefined filter was encountered. I checked and it seems to be NC_ENOFILTER error I believe. I also believe that this error stems from the. nc_get_vara_float() function. I ran the pres_temp_4D_rd.c example with no error similar to this.

I attempted to check the filters of the .nc file with:
ncdump -h | grep FILTERS

Nothing came up for filters in the ncdump. I also tried:
ncdump -s

I did not see a "chunking" section but did see a variable like this "ChunkSizes = 1, 1, 45, 57 ; for my pressure data variable.

Any reason that this is happening ? Can't seem to find much info on this issue.

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Feb 23, 2023 via email

@tayloj25
Copy link
Author

Nothing comes from fgrep -I filter, omitting this, I get this:

netcdf pressure_data {
dimensions:
time = 1 ;
z = 1 ;
y = 45 ;
x = 57 ;
variables:
float pressure(time, z, y, x) ;
pressure:valid_min = 94849.89f ;
pressure:valid_max = 107875.9f ;
pressure:_FillValue = -9999.f ;
pressure:long_name = "Pressure reduced to MSL" ;
pressure:units = "Pa" ;
pressure:grid_mapping = "grid_mapping" ;
pressure:_Storage = "chunked" ;
pressure:_ChunkSizes = 1, 1, 45, 57 ;
pressure:_DeflateLevel = 5 ;
pressure:_Endianness = "little" ;
double forecast_period(time) ;
forecast_period:standard_name = "forecast_period" ;
forecast_period:long_name = "time interval between the forecast reference time and the valid time" ;
forecast_period:units = "seconds" ;
forecast_period:_Storage = "contiguous" ;
forecast_period:_Endianness = "little" ;
double forecast_reference_time ;
forecast_reference_time:standard_name = "forecast_reference_time" ;
forecast_reference_time:long_name = "the time of the analysis from which the forecast was made" ;
forecast_reference_time:units = "seconds since 1970-01-01T00:00:00Z" ;
forecast_reference_time:calendar = "gregorian" ;
forecast_reference_time:reference_date = "2021-04-09T06:00:00Z" ;
forecast_reference_time:_Storage = "contiguous" ;
forecast_reference_time:_Endianness = "little" ;
int grid_mapping ;
grid_mapping:earth_radius = 6371229.f ;
grid_mapping:grid_mapping_name = "latitude_longitude" ;
grid_mapping:_Storage = "contiguous" ;
grid_mapping:_Endianness = "little" ;
double time(time) ;
time:standard_name = "time" ;
time:long_name = "data valid time" ;
time:units = "seconds since 1970-01-01T00:00:00Z" ;
time:calendar = "gregorian" ;
time:axis = "T" ;
time:reference_date = "2021-04-09T07:00:00Z" ;
time:_Storage = "contiguous" ;
time:_Endianness = "little" ;
float x(x) ;
x:standard_name = "longitude" ;
x:long_name = "longitude" ;
x:units = "degrees_east" ;
x:axis = "X" ;
x:_Storage = "contiguous" ;
x:_Endianness = "little" ;
float y(y) ;
y:standard_name = "latitude" ;
y:long_name = "latitude" ;
y:units = "degrees_north" ;
y:axis = "Y" ;
y:_Storage = "contiguous" ;
y:_Endianness = "little" ;
float z(z) ;
z:long_name = "Mean sea level" ;
z:axis = "Z" ;
z:_Storage = "contiguous" ;
z:_Endianness = "little" ;

// global attributes:
:Conventions = "CF-1.6" ;
:source = "US NWS - NCEP" ;
:title = "Global Forecast System Model" ;
:history = "Tue Apr 13 15:33:28 2021: ncks -d x,384,496,2 -d y,464,552,2 PRESSURE_DATA.nc pressure_data.nc\n",
"Converted to NetCDF from Grib2 with Grib2toNc, V1.25, from NCAR-RAL." ;
:NCO = "4.7.2" ;
:_NCProperties = "version=1|netcdflibversion=4.6.0|hdf5libversion=1.10.0" ;
:_SuperblockVersion = 0 ;
:_IsNetcdf4 = 0 ;
:_Format = "netCDF-4 classic model" ;
}

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Feb 23, 2023 via email

@tayloj25
Copy link
Author

I thought that I had already installed zlib.
I ran "nc-config --all" and got:

This netCDF 4.9.1 has been built with the following features:

--cc -> gcc
--cflags -> -I/usr/local/include -I/usr/local/include
--libs -> -L/usr/local/lib -lnetcdf
--static -> -lhdf5_hl -lhdf5 -lm -lz -ldl -lxml2 -lcurl

--has-c++ -> no
--cxx ->

--has-c++4 -> no
--cxx4 ->

--has-fortran -> no
--has-dap -> no
--has-dap2 -> no
--has-dap4 -> no
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> no
--has-logging -> no
--has-pnetcdf -> no
--has-szlib -> no
--has-cdf5 -> yes
--has-parallel4 -> no
--has-parallel -> no
--has-nczarr -> yes
--has-zstd -> no
--has-benchmarks -> no
--has-multifilters -> yes
--has-stdfilters -> deflate bz2
--has-quantize -> yes

--prefix -> /usr/local
--includedir -> /usr/local/include
--libdir -> /usr/local/lib
--plugindir ->
--version -> netCDF 4.9.1

Does the -lz in static mean that is included or do I need to add something else to these options?

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Feb 23, 2023 via email

@tayloj25
Copy link
Author

I thought I had, but I just ran it again and encountered failure with:
tst_netcdf4.sh
tst_ncgen4.sh
tst_netcdf4_4.sh
tst_nccopy4.sh
tst_nccopy5.sh

I have attached the test-suite.log below:

netCDF 4.9.1: ncdump/test-suite.log

TOTAL: 47

PASS: 42

SKIP: 0

XFAIL: 0

FAIL: 5

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: tst_netcdf4

*** Testing ncgen and ncdump for netCDF-4 format.
*** Creating test output tst_netcdf4_c0.nc.
*** Testing that program tst_h_rdc0 can read tst_netcdf4_c0.nc.

*** Checking HDF5 file c0.nc.
*** Checking HDF5 objcts...ok.
*** Tests successful!
*** Running tst_create_files.c to create test files.

*** Creating test files for ncdump.
*** creating nested group file tst_solar_1.nc...ok.
*** checking nested group file tst_solar_1.nc...ok.
*** creating file with VLEN tst_solar_2.nc...ok.
*** creating file with compound type tst_solar_cmp.nc...ok.
*** Tests successful!
*** Testing tst_create_files output for netCDF-4 features.
*** Running tst_group_data.c to create test files.

*** Testing groups.
*** creating nested group file tst_group_data.nc...ok.
*** Tests successful!
*** Testing -v option with absolute name and groups...
*** Testing -v option with relative name and groups...
*** Running tst_enum_data.c to create test files.

*** Testing enums.
*** creating enum test file tst_enum_data.nc...ok.
*** Tests successful!
*** Running tst_enum_undef.c to create test files.

*** Testing enum undefined identifier.
*** creating enum test file tst_enum_undef.nc...ok.
*** Tests successful!
*** Running tst_opaque_data.c to create test files.

*** Testing opaque types.
*** creating opaque test file tst_opaque_data.nc...ok.
*** Tests successful!
*** Running tst_comp.c to create test files.

*** Testing compound types.
*** creating compound test file tst_comp.nc...ok.
*** Tests successful!
*** Running tst_nans.c to create test files.

*** Testing NaN
*** creating NaN test file tst_nans.nc...ok.
*** Tests successful!
*** Running tst_special_atts.c to create test files.

*** Testing '-s' option for special attributes.
*** creating special attributes test file tst_special_atts.nc...Sorry! Unexpected result, tst_special_atts.c, line: 130
FAIL tst_netcdf4.sh (exit status: 2)

FAIL: tst_ncgen4

*** Performing diff tests: k=3
*** Testing ncgen with -k3
*** Testing: nc_enddef
*** SUCCEED: nc_enddef
*** Testing: ref_tst_unicode
*** SUCCEED: ref_tst_unicode
*** Testing: ref_tst_utf8
*** SUCCEED: ref_tst_utf8
*** Testing: simple_xy
*** SUCCEED: simple_xy
*** Testing: small
*** SUCCEED: small
*** Testing: nc_sync
*** SUCCEED: nc_sync
*** Testing: ref_tst_small
*** SUCCEED: ref_tst_small
*** Testing: small2
*** SUCCEED: small2
*** Testing: tst_ncml
*** SUCCEED: tst_ncml
*** Testing: n3time
*** SUCCEED: n3time
*** Testing: ref_tst_chardata
*** XFAIL : ref_tst_chardata
*** Testing: ref_tst_nul3
*** SUCCEED: ref_tst_nul3
*** Testing: ref_tst_long_charconst
*** SUCCEED: ref_tst_long_charconst
*** Testing: tst_chararray
*** SUCCEED: tst_chararray
*** Testing: unlimtest1
*** SUCCEED: unlimtest1
*** Testing: ref_keyword
*** SUCCEED: ref_keyword
*** Testing: test0
*** SUCCEED: test0
*** Testing: sfc_pres_temp
*** SUCCEED: sfc_pres_temp
*** Testing: fills
*** SUCCEED: fills
*** Testing: c0
*** SUCCEED: c0
*** Testing: example_good
*** SUCCEED: example_good
*** Testing: pres_temp_4D
*** SUCCEED: pres_temp_4D
*** Testing: ref_nctst
*** SUCCEED: ref_nctst
*** Testing: ref_nctst_64bit_offset
*** SUCCEED: ref_nctst_64bit_offset
*** Testing: ref_ctest1_nc4
*** SUCCEED: ref_ctest1_nc4
*** Testing: ref_ctest1_nc4c
*** SUCCEED: ref_ctest1_nc4c
*** Testing: ref_nctst_netcdf4
*** SUCCEED: ref_nctst_netcdf4
*** Testing: ref_nctst_netcdf4_classic
*** SUCCEED: ref_nctst_netcdf4_classic
*** Testing: ref_tst_unlim2
*** XFAIL : ref_tst_unlim2
*** Testing: ref_tst_names
*** SUCCEED: ref_tst_names
*** Testing: ref_dimscope
*** SUCCEED: ref_dimscope
*** Testing: ref_typescope
*** SUCCEED: ref_typescope
*** Testing: ref_tst_string_data
*** SUCCEED: ref_tst_string_data
*** Testing: ref_tst_comp
*** SUCCEED: ref_tst_comp
*** Testing: ref_tst_comp2
*** SUCCEED: ref_tst_comp2
*** Testing: ref_tst_comp3
*** SUCCEED: ref_tst_comp3
*** Testing: ref_tst_group_data
*** SUCCEED: ref_tst_group_data
*** Testing: ref_tst_opaque_data
*** SUCCEED: ref_tst_opaque_data
*** Testing: ref_tst_solar_1
*** SUCCEED: ref_tst_solar_1
*** Testing: ref_tst_solar_2
*** SUCCEED: ref_tst_solar_2
*** Testing: ref_tst_enum_data
*** SUCCEED: ref_tst_enum_data
*** Testing: ref_tst_special_atts
ncgen: NetCDF: Filter error: undefined filter encountered
(genbin.c:genbin_definespecialattributes:231)
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: H5PLint.c line 261 in H5PL_load(): search in path table failed
major: Plugin for dynamically loaded library
minor: Can't get value
#1: H5PLpath.c line 802 in H5PL__find_plugin_in_path_table(): search in path /usr/local/hdf5/lib/plugin encountered an error
major: Plugin for dynamically loaded library
minor: Can't get value
#2: H5PLpath.c line 855 in H5PL__find_plugin_in_path(): can't open directory: /usr/local/hdf5/lib/plugin
major: Plugin for dynamically loaded library
minor: Can't open directory or file
FAIL tst_ncgen4.sh (exit status: 1)

FAIL: tst_netcdf4_4

*** Running extra netcdf-4 tests.
*** running tst_string_data to create test files...

*** Testing strings.
*** creating strings test file tst_string_data.nc...ok.
*** Tests successful!
*** dumping tst_string_data.nc to tst_string_data.cdl...
*** comparing tst_string_data.cdl with ref_tst_string_data.cdl...
*** testing reference file ref_tst_compounds2.nc...
*** testing reference file ref_tst_compounds3.nc...
*** testing reference file ref_tst_compounds4.nc...
Filter id=1; filter not available
ncgen: NetCDF: Filter error: undefined filter encountered
(genbin.c:genbin_definespecialattributes:259)
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: H5PLint.c line 261 in H5PL_load(): search in path table failed
major: Plugin for dynamically loaded library
minor: Can't get value
#1: H5PLpath.c line 802 in H5PL__find_plugin_in_path_table(): search in path /usr/local/hdf5/lib/plugin encountered an error
major: Plugin for dynamically loaded library
minor: Can't get value
#2: H5PLpath.c line 855 in H5PL__find_plugin_in_path(): can't open directory: /usr/local/hdf5/lib/plugin
major: Plugin for dynamically loaded library
minor: Can't open directory or file
FAIL tst_netcdf4_4.sh (exit status: 1)

FAIL: tst_nccopy4

*** Testing groups.
*** creating nested group file tst_group_data.nc...ok.
*** Tests successful!

*** Testing enums.
*** creating enum test file tst_enum_data.nc...ok.
*** Tests successful!

*** Testing compound types.
*** creating compound test file tst_comp.nc...ok.
*** Tests successful!

*** Testing compound types some more.
*** creating another compound test file tst_comp2.nc...ok.
*** Tests successful!

*** Testing netCDF-4 features of nccopy on ncdump/*.nc files
*** Test nccopy tst_comp.nc copy_of_tst_comp.nc ...
*** compare with copy_of_tst_comp.cdl
*** Test nccopy tst_comp2.nc copy_of_tst_comp2.nc ...
*** compare with copy_of_tst_comp2.cdl
*** Test nccopy tst_enum_data.nc copy_of_tst_enum_data.nc ...
*** compare with copy_of_tst_enum_data.cdl
*** Test nccopy tst_fillbug.nc copy_of_tst_fillbug.nc ...
*** compare with copy_of_tst_fillbug.cdl
*** Test nccopy tst_group_data.nc copy_of_tst_group_data.nc ...
*** compare with copy_of_tst_group_data.cdl
*** Test nccopy tst_nans.nc copy_of_tst_nans.nc ...
*** compare with copy_of_tst_nans.cdl
*** Test nccopy tst_opaque_data.nc copy_of_tst_opaque_data.nc ...
*** compare with copy_of_tst_opaque_data.cdl
*** Test nccopy tst_solar_1.nc copy_of_tst_solar_1.nc ...
*** compare with copy_of_tst_solar_1.cdl
*** Test nccopy tst_solar_2.nc copy_of_tst_solar_2.nc ...
*** compare with copy_of_tst_solar_2.cdl
*** Test nccopy tst_solar_cmp.nc copy_of_tst_solar_cmp.nc ...
*** compare with copy_of_tst_solar_cmp.cdl
*** Test nccopy tst_special_atts.nc copy_of_tst_special_atts.nc ...
*** compare with copy_of_tst_special_atts.cdl
*** Test nccopy tst_string_data.nc copy_of_tst_string_data.nc ...
*** compare with copy_of_tst_string_data.cdl
*** Creating compressible test files tst_inflated.nc, tst_inflated4.nc...ok.
*** Tests successful!
*** Test nccopy -d1 can compress a classic format file ...
NetCDF: Filter error: undefined filter encountered
Location: file nccopy.c; fcn copy_var_specials line 1152
FAIL tst_nccopy4.sh (exit status: 1)

FAIL: tst_nccopy5

*** Creating chunkable test file tmp_nc5_base.nc...
ok.
*** Tests successful!
*** Test nccopy -c with per-variable chunking; classic->enhanced
*** Test nccopy -c with per-variable chunking; enhanced->enhanced
*** Creating chunkable test file tst_nc5.nc...
option: deflate
Sorry! Unexpected result(0), tst_chunking.c, line: 103
1 failures
1 errors detected! Sorry!
FAIL tst_nccopy5.sh (exit status: 2)

I'm guessing that it has to do with that last error? Any ideas ?

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Feb 23, 2023 via email

@tayloj25
Copy link
Author

CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure --prefix=/usr/local --disable-dap --enable-byterange

I believe that is it... Is that what you were requesting? Is there some way to check it? I also may have added --disable-byterange at the end because I was getting a configure error: curl required for byte range support.

I installed curl and I believe that fixed the error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants