CICE PIO w/ HDF5 fails to write output via link

[aerorahul]

Description

The UFS-weather-model PR #2145 recently added control for PIO namelist in CICE. The update to CICE, specifically in ice_pio.F90 exposed an issue writing the output via links when using hdf5 for CICE history and restart formats. This was discovered in the global-workflow on WCOSS2, when the hash of the ufs-weather-model was attempted to be updated. See discussion in NOAA-EMC/global-workflow#2505

Thanks to @DeniseWorthen for tracing the source of the issue and working with developers to provide an update that will re-enable writing CICE output (and restarts) via links when hdf5 is used.

HDF5 is being used on WCOSS2, because it does not have the correct version of the pnetcdf library. An update is expected soon for WCOSS2 and pnetcdf.
This error was reproduced on Hera by choosing hdf5.

To Reproduce:

• Create a sandbox
• add links to CICE output in CICE_OUTPUT/ and CICE_RESTART/ for the expected output files.
• run the model.

An error such as below will be experienced:

21:  (abort_ice)ABORTED:
21:  (abort_ice) called from ice_pio.F90
21:  (abort_ice) line number          223
21:  (abort_ice) error =
21:  (ice_pio_check)Permission denied, (ice_pio_init) ERROR: Failed to create file .
21:  /CICE_OUTPUT/iceh_ic.2021-03-23-43200.nc

Additional context

See global-workflow PR NOAA-EMC/global-workflow#2505

Output

https://gist.github.com/WalterKolczynski-NOAA/54d183f100ffcbac2786d393766c490c

[DeniseWorthen]

The rebuilt pio is now available on Cactus (#2232). If it resolves the issue w/ using pnetcdf, a UWM hotfix will be required to revert the WCOSS2 work-around in default_vars. The need to use hdf5 on WCOSS2 is breaking the G-W. @BrianCurtis-NOAA @jkbk2004

[DeniseWorthen]

@aerorahul I was provided an interesting link from one of our CICE colleagues open-mpi/ompi#12141. I don't follow all the details in that issue or how relevant it is to our case.