in 4.7.2, parallel I/O not working

[edhartnett]

I just noticed that 4.7.2 netcdf_par.h includes ncexternl.h, but this is not installed.

This seems to be fixed in master, but I believe it means that 4.7.2 will not work for parallel I/O, which is a fairly serious problem!

Or am I mistaken?

[ckhroulev]

This is correct (see #1502). NetCDF 4.7.2 cannot be used for parallel I/O, which forced me to downgrade to 4.7.0 since 4.7.1 does not work on macOS.

[edhartnett]

OK, bummer. @WardF is the plan to get the next release out soon? Preferably before the AGU?

[edhartnett]

@WardF given that parallel I/O is a major feature, and one that NOAA and other HPC users absolutely need, can we soon get a new release where parallel I/O is working?

Thanks

[junwang-noaa]

May I ask if this parallel I/O is working for compressed netcdf4(HDF5) data? If yes, what is the time line for the release package ready? We are working on the next GFS implementation, it is planned to have the model output history files in compressed netcdf4 format, but currently downstream jobs are suffering the slowness of reading the data. We hope the parallel I/O can help resolving the issue. Thanks.

[edwardhartnett]

Howdy @junwang-noaa parallel I/O can read but not write compressed data. That will be the state of the next release.

However, the HDF5 team has created a filter that allows compression when writing with parallel I/O and they have promised to help me get this going soon after the new year.

But I would like to take a look at one of your history files to see if I can understand what is going on. Can you share one with me?

[junwang-noaa]

I have two sample data on EMC ftp site at:

https://ftp.emc.ncep.noaa.gov/EIB/sample/gfsv16_compressed_netcdf/gfs.t00z.atmf024.nc
https://ftp.emc.ncep.noaa.gov/EIB/sample/gfsv16_compressed_netcdf/gfs.t00z.sfcf024.nc

The atmf file is compressed and the sfcf is not. The sequential read of the two files takes about 5 minuts, while previous parallel read of the plain binary files takes 20s.

[edwardhartnett]

Thanks I will take a look...

[edwardhartnett]

Let's continue this discussion in #1543

[edhartnett]

The parallel I/O build issue has been fixed by the 4.7.3 release, so I will close this issue.