RuntimeError: NetCDF: HDF error

[DEHewitt]

Hi,

I am having some trouble running simulations for one year of the the 10-year oceanographic model that I have (daily outputs). The simulation appears to run for a little while

INFO: Compiled random ==> /scratch/pbs.1186449[136].kman.restech.unsw.edu.au/parcels-15278054/parcels_random_e6ee00a3-2127-41b4-9bb0-4c81c7aa0c4e.so
INFO: Compiled SampleParticleSampleInitialAdvectionRK4_3D_alternativeSampleAgeSampleDistanceDiffusionUniformKhSampleTempSampleBathyResetDepthSampleParticleDepthlarvalBuoyancySampleVelocitiesUnbeaching ==> /scratch/pbs.1186449[136].kman.restech.unsw.edu.au/parcels-15278054/b88a7ab377d8d6399793821c889b487f_0.so
WARNING: dt or runtime are zero, or endtime is equal to Particle.time. The kernels will be executed once, without incrementing time
  0% |                                                                        |
  1% |                                                                        |
  2% |#                                                                       |
WARNING: Deleting particle because of bug in #749 and #737
  3% |##                                                                      |
  4% |###                                                                     |
  6% |####                                                                    |
  7% |#####                                                                   |
  8% |######                                                                  |
  9% |#######                                                                 |
 10% |#######                                                                 |
 12% |########                                                                |
 13% |#########                                                               |
 14% |##########                                                              |
 15% |###########                                                             |
 17% |############                                                            |
 18% |#############                                                           |
 19% |##############                                                          |
 20% |##############                                                          |
 21% |###############                                                         |
 23% |################                                                        |
 24% |#################                                                       |
 25% |##################                                                      |
 26% |###################                                                     |
 28% |####################                                                    |
 29% |#####################                                                   |
 30% |#####################                                                   |
 31% |######################                                                  |

Then I get the following:

HDF5-DIAG: Error detected in HDF5 (1.10.5) thread 46940387305216:
  #000: H5Dio.c line 199 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: H5Dio.c line 601 in H5D__read(): can't read data
    major: Dataset
    minor: Read failed
  #002: H5Dchunk.c line 2259 in H5D__chunk_read(): unable to read raw data chunk
    major: Low-level I/O
    minor: Read failed
  #003: H5Dchunk.c line 3624 in H5D__chunk_lock(): data pipeline read failed
    major: Dataset
    minor: Filter operation failed
  #004: H5Z.c line 1326 in H5Z_pipeline(): filter returned failure during read
    major: Data filters
    minor: Read failed
  #005: H5Zdeflate.c line 123 in H5Z_filter_deflate(): inflate() failed
    major: Data filters
    minor: Unable to initialize object

followed by a more familiar Python error:

Traceback (most recent call last):
  File "portunid_particle_tracking.py", line 350, in <module>
    recovery={ErrorCode.ErrorOutOfBounds: DeleteParticle})
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/parcels/particleset.py", line 642, in execute
    execute_once=execute_once)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/parcels/kernel.py", line 431, in execute
    self.execute_jit(pset, endtime, dt)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/parcels/kernel.py", line 245, in execute_jit
    f.chunk_data()
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/parcels/field.py", line 1089, in chunk_data
    self.data_chunks[block_id] = np.array(self.data.blocks[(slice(self.grid.tdim),) + block])
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/array/core.py", line 1365, in __array__
    x = self.compute()
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/threaded.py", line 84, in get
    **kwargs
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/dask/array/core.py", line 102, in getter
    c = np.asarray(c)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/core/indexing.py", line 495, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/core/indexing.py", line 655, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/core/indexing.py", line 560, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 73, in __getitem__
    key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/core/indexing.py", line 845, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/home/z5278054/python_parcels/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 85, in _getitem
    array = getitem(original_array, key)
  File "netCDF4/_netCDF4.pyx", line 4452, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 5394, in netCDF4._netCDF4.Variable._get
  File "netCDF4/_netCDF4.pyx", line 1928, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

I suspect the initial error (the HDF5-DIAG error) is causing the second error, as my simulations for other years work as expected. I have done a few searches but the returned errors are beyond my knowledge.

Any help would be so great!

Thanks

[erikvansebille]

Hi @DEHewitt, these errors normally occur if there's something wrong with the netcdf file and xarray is unable to properly open them. Can you check if the netcdf file that is being read at that point in the simulation is not corrupt?

[DEHewitt]

Hi @erikvansebille - looks like it was a corrupt file. Thanks!

[DEHewitt]

Hi @erikvansebille, I am returning to this issue as I have encountered it again. However, now it is for a different file (day). Curiously, I am now only getting the error for backwards simulations (forwards simulations over the same date range run without issue).

I have pinpointed when the simulation crashes, and as a troubleshooting measure attempted to open the suspect file(s) with a variety of functions (e.g., xarray.open_dataset() and ncdf4::nc_open() in R) which all work, to check if it was an issue with corruption (as above). Is there another explanation for this? I don't think it is an issue with the rest of the code, as simulations for other years run and output is as expected.

[CKehl]

well, the HDF errors are pretty on-spot: seems your NetCDF files are compressed (see the inflate() error which refers to the inlfate function of the zip-algorithm). In fact, coupling dask chunking with compressed data isn't advisable, because data compression (which needs global data and statistics for compression) interferes with local reads.

a) is it possible / true that your data are compressed ?
b) if so: can you store your data in an uncompressed format, and retry the simulation again ? Can you post the log for that run then ?
c) if (a) is true and (b) results in 'no', then you need to define field = FieldSet.from_netcdf(..., chunksize=None) when creating the fieldset. This obviously results in the fact that your computing machine needs to be able to hold all the data for 2 timesteps in memory, but that's the tradeoff required then here.

[DEHewitt]

Thanks @CKehl - I haven't forgotten to reply to this. Will just be a little while before I get back to this.

[DEHewitt]

Hi @CKehl,

It's taken me a little while to get back to this. To answer your questions:
a) they were shared in compressed (zipped) folders and unzipped (by me) once uploaded to the HPC. I haven't been able to find any straight forward way to check if they're compressed or not, but the fact that the same files work for forwards simulations leads me to expect this isn't the issue.
b) if it turns out they are compressed I certainly can, and will
c) noted, and this should be possible

As an alternative, is it possible to remove the troublesome file and run the simulation with FIeldSet.from_nemo(allow_time_extrapolation = True) and skip over that day? It is a 10-year simulation (~ 200 particles release days per year) interested in broad patterns of larval dispersal and I don't think that skipping one day is going to particularly detrimental to the patterns. I just wanted to confirm that this would have the effect I suspect, that is, it'd use the velocity fields of the day previous where the file is missing, and then on the next day where there is a file it'd advect using those correct fields.

Thanks for your help.

[erikvansebille]

Hi @DEHewitt, I'm just quickly going to chime in here because I think you misunderstand what allow_time_extrapolation = True does. This is to extrapolate the hydrodynamic data (i.e. allow particles to move beyond the time available in your dataset, by simply setting the flow field constant).

If you remove one file from within the series of snapshots, then you don't need to change anything to your code. Parcels does not assume or require a constant interval between hydrodynamic snapshots, instead it will simply interpolate linearly between two consecutive snapshots. So if one at time tⁿ is 'missing', it will simply interpolate between the snapshots at time t^n-1 and tⁿ⁺¹

[DEHewitt]

Thanks @erikvansebille, I might've made it unclear by saying "skip over that day", but what you describe is conceptually what I meant (although I didn't realize parcels would linearly interpolate, rather I thought it'd use the most recent snapshot, i.e., t^n-1).