When esmf 8.4.0 is used coupled configuration tests fail

[DusanJovic-NOAA]

Description

ESMF version 8.4.0 (and I believe 8.3.0 release but not 8.3.0b09) uses updated PIO library which has stricter attribute type checks, which causes model to fail due to mismatched floating point type (float vs double) used for some attributes (_FillValue and missing_value).

To Reproduce:

Build ESMF 8.4.0, use it to build the model and run one of the coupled regression tests, for example cpld_control_p8. It will fail with this error:

20230111 163655.058 ERROR            PET144 ESMCI_PIO_Handler.C:1184 ESMCI::PIO_Handler::arrayWriteOn The NetCDF Library returned an error  -  Attempting to end definition of variable: bc1, (PIO/NetCDF error = NetCDF: Not a valid data type or _FillValue type mismatch)
20230111 163655.059 ERROR            PET144 ESMCI_IO_Handler.C:455 ESMCI::IO_Handler::arrayWrite() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 ESMCI_IO.C:723 ESMCI::IO::write() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 ESMCI_IO.C:494 ESMCI::IO::write() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 ESMCI_IO_F.C:171 c_esmc_iowrite() Unable to write to file  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 ESMF_IO.F90:523 ESMF_IOAddArray() Unable to write to file  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 ESMF_FieldBundle.F90:17681 ESMF_FieldBundleWrite() Unable to write to file  - Internal subroutine call returned Error
20230111 163655.059 ERROR            PET144 module_wrt_grid_comp.F90:3388 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 module_wrt_grid_comp.F90:2951 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 module_wrt_grid_comp.F90:2117 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 fv3_cap.F90:1124 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 ATM:src/addon/NUOPC/src/NUOPC_ModelBase.F90:2207 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3488 Unable to write to file  - Phase 'phase2' Run for modelComp 2 did not return ESMF_SUCCESS
20230111 163655.059 ERROR            PET144 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3726 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3403 Unable to write to file  - Passing error in return code
20230111 163655.059 ERROR            PET144 UFS.F90:403 Unable to write to file  - Aborting UFS
20230111 163655.059 INFO             PET144 Finalizing ESMF

Additional context

Output

[DeniseWorthen]

The bc1 field seems to be an aerosol field (?) so why doesn't the atmaero_control_p8 test also fail?

[DusanJovic-NOAA]

The bc1 field seems to be an aerosol field (?) so why doesn't the atmaero_control_p8 test also fail?

Let me double check.

[DusanJovic-NOAA]

The bc1 field seems to be an aerosol field (?) so why doesn't the atmaero_control_p8 test also fail?

Let me double check.

atmaero_control_p8 test is using the write grid component which is not using ESMF I/O (which uses PIO).

[DeniseWorthen]

I should know this part better than I do. Does using wrt_grd come down to whether you want your output grid to be cubed-sphere vs gaussian etc?

[jkbk2004]

@DusanJovic-NOAA should we try 8.4.1 (currently 8.4.1b01, I guess)? I thought it might have a fix.

[DusanJovic-NOAA]

I should know this part better than I do. Does using wrt_grd come down to whether you want your output grid to be cubed-sphere vs gaussian etc?

The write grid comp is used for all history output grids but 'multi-tile' cubed sphere grid (cubed sphere grid where each tile is in a separate file) is written out by ESMF I/O (ie. PIO), while other output grids (gaussian, lambert, lat-lon, rotated lat-lon) are written by module_write_netcdf.

[DusanJovic-NOAA]

@DusanJovic-NOAA should we try 8.4.1 (currently 8.4.1b01, I guess)? I thought it might have a fix.

The problem is not in ESMF but in our code, where we must explicitly set the values of these two attributes (_FillValue and missing_value) to be the same kind as actual data values. See my 'missing_value_r4' FV3 and atmos_cubed_sphere branches. https://github.com/DusanJovic-NOAA/ufs-weather-model/tree/missing_value_r4

I'm still running some tests. I'll open PR soon.