Runtime segmentation fault using conda-forge netcdf4

[ngoldbaum]

When I try to run the following script using conda-forge's linux-64 netcdf4 1.3.1 build (1.3.1-py36_2):

>>> from netCDF4 import Dataset
>>> Dataset('galaxy_hdf5_plt_cnt_0000')

I get the following error:

python: /feedstock_root/build_artefacts/libnetcdf_1510148087952/work/netcdf-c-4.5.0/libsrc4/nc4var.c:367: nc4_vararray_add: Assertion `var->varid == grp->vars.nelems' failed.

The same does not happen with the netcdf4 1.3.1 build from defaults (1.3.1-py36hfd655bd_2).

The file is available here (warning, 3 GB): http://use.yt/upload/2d8304c6

You can download it like this: curl -JO http://use.yt/upload/2d8304c6, which will save a file named memory_error_example_yt.tar.

Unfortunately I did not produce this file, it was produced by a yt user who reported a different error but hapenned to run into this netcdf4 error while trying to reproduce their original error.

[ocefpaf]

@dopplershift do you have an idea of what may be going on here?

[dopplershift]

Not really. @ngoldbaum are we sure this file isn't corrupted in some way outside of a library problem?

[ngoldbaum]

@dopplershift I don't think so?

It's not even a NetCDF file, it's just a plain old HDF5 file generated by the FLASH simulation code, although for some reason NetCDF parses it as a NetCDF file using the version of NetCDF4 in the defaults channel.

yt has a load() function that attempts to automatically determine whether a given random file is of a format that yt supports out of the box. To do this it has a number of heuristics. The one for NetCDF4 files looks like:

def _is_valid(path)
    try:
        with Dataset(path) as f:
            # look inside f for structure that corresponds to the file type we want to load
            if hueristics_match:
                return True
    except Exception as e:
        pass
    return False

If it's not a valid NetCD4 file, I'd argue NetCDF4 should always raise an error of some sort. In this case it's raising an error at the C level, which is crashing the whole python interpreter, which isn't very friendly.

[dopplershift]

@WardF Does the netCDF-C library do any checks on a file to ensure it's valid netCDF and not random HDF5?

[ngoldbaum]

Regardless of the answer to that question, it seems odd to me that different builds of the same library version will either crash or parse the file. Perhaps there’s some additional debug validation that the conda-forge version is doing that the defaults version isn’t?

[isuruf]

Regardless of the answer to that question, it seems odd to me that different builds of the same library version will either crash or parse the file.

It's an assertion that fails. libnetcdf-feedstock builds a debug version of the library (@conda-forge/libnetcdf, why?), and therefore the assertion fails. Defaults is building the release version and therefore the assertion is simply skipped.

[ocefpaf]

libnetcdf-feedstock builds a debug version of the library

Do we? Not sure what triggers that in the build.

[isuruf]

Do we?

Some CMake projects have Debug as the default, some have Release as the default. Add -DCMAKE_BUILD_TYPE=Release to make sure it's a Release build.

[ocefpaf]

Thanks! Doing that now...

[isuruf]

Btw, this would only be hiding an internal error, not a fix.

[ngoldbaum]

Yes, we've certainly run into other cases where we hand NetCDF4 an hdf5 file that isn't a NetCDF4 file but it still tries to parse anyway. See e.g. Unidata/netcdf-c#320

Ideally there'd be a better way to determine whether an arbitrary HDF5 file is a NetCDF4 file besides trying to load it, but I'm not aware of such a simple check.

[jjhelmus]

Ideally there'd be a better way to determine whether an arbitrary HDF5 file is a NetCDF4 file besides trying to load it, but I'm not aware of such a simple check.

All NetCDF4 classic model files should have the attribute _nc3_strict in the root group. Unfortunately non-classic format file explicitly do not have this attribute.

[isuruf]

Ideally there'd be a better way to determine whether an arbitrary HDF5 file is a NetCDF4 file besides trying to load it, but I'm not aware of such a simple check.

Ah, I wasn't following the full thread. Now it makes sense why an assert might fail. (Because you are asking it to parse something that it doesn't understand.)