Using a miniforge environment, iris can load data from .nc file but with a lot of errors

[penmaher]

📰 Custom Issue

I have found that miniforge is unable to load an iris cube without errors but interestingly miniconda can. I thought I would raise this as an issue, though it is not explicitly an iris code issue but a python environment/library dependency issue.

A minimum working example is as follows:

import iris
import glob
filename = iris.sample_data_path('ostia_monthly.nc')
f1 = glob.glob(filename)
cube = iris.load_cube(f1)
var = cube.data

When the above code is run using a miniforge environment (see py3.10 env), the error attached in the txt file is generated
error.txt. But when I run the same code using a minconda environment (see py3.10conda env) the code executes without error.

For context, here is the python environments I have been working with (using the Ubuntu 22.04 OS).

1. Working in a miniforge environment at version 22.9.0-2 which was set up using the following:

   https://github.com/conda-forge/miniforge/releases/download/22.9.0-2/Miniforge-pypy3-Linux-x86_64.sh

2. Created a python environment at version 3.10:

   conda create -n py3.10 python=3.10

   conda install cube_helper iris pandas

What did work, was using a miniconda environment instead:

1. Working in a miniconda environment at version 23.1.0-1 which was set up using the following:

https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh

2. create a python environment at version 3.10

   conda create -n py3.10conda python=3.10

   conda install pandas

   conda install -c conda-forge cube_helper iris

[pp-mo]

For ref. the errors in the file look like this ...

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 476 in H5O__attr_open_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #006: H5Adense.c line 394 in H5A__dense_open(): can't locate attribute in name index
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
 . . .
  ((and lots more similar))

[pp-mo]

This looks familiar : It is a current known problem / ongoing investigation, which appears to occur with later versions of libnetcdf / netCDF4 / hdf5.
Details : #5187

For now, we think the messages are effectively benign -- the operations themselves seem to complete successfully.
( However, the messages themselves do seem to indicate actual errors in the dask workers ? )

The best workaround for now seems to be, to pin "libnetcdf < 4.9" as here.

[HGWright]

@SciTools/peloton We believe pinning "libnetcdf < 4.9" has solved this for now. We are going to close the issue as it no longer is an issue. If this becomes relevant again then feel free to reopen.