hafs_3denvar_glbens ctest fails on Hercules

[DavidHuber-NOAA]

The hafs_3denvar_glbens ctest is failing to produce identical fv3_dynvars files between the hafs_3denvar_glbens_loproc_updat and hafs_3denvar_glbens_hiproc_updat test cases. This was discovered during testing of #684 and #695.

[TingLei-NOAA]

@DavidHuber-NOAA I tried looking into this starting from GSI compiled with debug mode. But seems GSI in their ctests would abort in the debug mode for issues to be fixed by #679. While I am not sure when that PR would be merged to the EMC GSI, is that possible you directly merge that branch to your GSI branch and point to me?

[DavidHuber-NOAA]

@TingLei-NOAA I would suggest doing that on a local clone since the PR may still change:

cd <gsi>
git remote add hafs https://github.com/hafs-community/gsi
git fetch hafs
git merge hafs/feature/toff_fix

[TingLei-NOAA]

@DavidHuber-NOAA But, which branch are you using now for your current spack setup?

[DavidHuber-NOAA]

Ah, right:

git clone https://github.com/DavidHuber-NOAA/GSI --recursive -b feature/ss_160 gsi_ss_160
cd gsi_ss_160
git remote add hafs https://github.com/hafs-community/gsi
git fetch hafs
git merge hafs/feature/toff_fix

[RussTreadon-NOAA]

While running ctests on Hercules for PR #684, failures occurred in the rrfs_3denvar_glbens and hafs_3denvar_hybens tests.

The rrfs_3denvar_glbens was due to the loproc and hiproc updat runs (i.e., the spack-stack/1.6.0 build) not producing identical fv3_dynvars. Comparison of fv3_dynvars records between the loproc and hiproc runs shows differences in the u and v wind components

hercules-login-3:/work2/noaa/stmp/rtreadon/pr684/tmpreg_rrfs_3denvar_glbens$ /work/noaa/da/rtreadon/bin/compare_ncfile.py rrfs_3denvar_glbens_loproc_updat/fv3_dynvars rrfs_3denvar_glbens_hiproc_updat/fv3_dynvars
/work/noaa/da/rtreadon/bin/compare_ncfile.py:12: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  data1 = nc1[varname][:]
/work/noaa/da/rtreadon/bin/compare_ncfile.py:13: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  data2 = nc2[varname][:]
xaxis_1 min/max 1=1.0,396.0 min/max 2=1.0,396.0 max abs diff=0.0000000000
xaxis_2 min/max 1=1.0,397.0 min/max 2=1.0,397.0 max abs diff=0.0000000000
yaxis_1 min/max 1=1.0,233.0 min/max 2=1.0,233.0 max abs diff=0.0000000000
yaxis_2 min/max 1=1.0,232.0 min/max 2=1.0,232.0 max abs diff=0.0000000000
zaxis_1 min/max 1=1.0,65.0 min/max 2=1.0,65.0 max abs diff=0.0000000000
Time min/max 1=1.0,1.0 min/max 2=1.0,1.0 max abs diff=0.0000000000
u min/max 1=-38.006363,59.550613 min/max 2=-38.006363,59.550613 max abs diff=0.0057044029
v min/max 1=-26.81582,31.66718 min/max 2=-26.81582,31.66718 max abs diff=10.2818336487
W min/max 1=-2.563452,6.3780456 min/max 2=-2.563452,6.3780456 max abs diff=0.0000000000
DZ min/max 1=-5746.5317,-17.513391 min/max 2=-5746.5317,-17.513391 max abs diff=0.0000000000
T min/max 1=194.30504,313.18134 min/max 2=194.30504,313.18134 max abs diff=0.0000000000
delp min/max 1=140.36311,3325.6877 min/max 2=140.36311,3325.6877 max abs diff=0.0000000000
phis min/max 1=-676.6589,36239.055 min/max 2=-676.6589,36239.055 max abs diff=0.0000000000

The hafs_3denvar_hybens failure is also due to differences between the loproc and_hiproc_ updat runs. However, for this case differences occur in the T (temperature) record

xaxis_1 min/max 1=1.0,720.0 min/max 2=1.0,720.0 max abs diff=0.0000000000
xaxis_2 min/max 1=1.0,721.0 min/max 2=1.0,721.0 max abs diff=0.0000000000
yaxis_1 min/max 1=1.0,541.0 min/max 2=1.0,541.0 max abs diff=0.0000000000
yaxis_2 min/max 1=1.0,540.0 min/max 2=1.0,540.0 max abs diff=0.0000000000
zaxis_1 min/max 1=1.0,65.0 min/max 2=1.0,65.0 max abs diff=0.0000000000
Time min/max 1=1.0,1.0 min/max 2=1.0,1.0 max abs diff=0.0000000000
u min/max 1=-49.416553,49.226555 min/max 2=-49.416553,49.226555 max abs diff=0.0000000000
v min/max 1=-49.340206,39.08488 min/max 2=-49.340206,39.08488 max abs diff=0.0000000000
W min/max 1=-2.842067,9.540336 min/max 2=-2.842067,9.540336 max abs diff=0.0000000000
DZ min/max 1=-5518.389,-17.457632 min/max 2=-5518.389,-17.457632 max abs diff=0.0000000000
T min/max 1=181.89748,346.99887 min/max 2=91.40897,346.99887 max abs diff=162.1640167236
delp min/max 1=140.48375,3329.2869 min/max 2=140.48375,3329.2869 max abs diff=0.0000000000
phis min/max 1=-6.5023405e-06,36102.867 min/max 2=-6.5023405e-06,36102.867 max abs diff=0.0000000000
ua min/max 1=-49.661697,46.939003 min/max 2=-49.661697,46.939003 max abs diff=0.0000000000
va min/max 1=-37.868477,49.013893 min/max 2=-37.868477,49.013893 max abs diff=0.0000000000

As a test, rerun rrfs_3denvar_glbens on Hercules. This time the test passed. The fv3_dynvars files are identical between updat and contrl and for loproc and hiproc.

Test project /work2/noaa/da/rtreadon/git/gsi/pr684/build
    Start 3: rrfs_3denvar_glbens
1/1 Test #3: rrfs_3denvar_glbens ..............   Passed  968.61 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 968.61 sec

[TingLei-NOAA]

An update : the branch incorporating changes in #679 still shows the issue. As @RussTreadon-NOAA found , the differences are in u and v, as shown below (for v, differences on 23 level).

Hence, based on the current findings (thanks to @RussTreadon-NOAA investigation on this issue), I agree, this issue should be caused by the issue #694 and #674, in which it was found when I_MPI_EXTRA_FILESYSTEM, the "hdf error" would occur in IO for u and v in the writing of the analysis. Now, alongside findings reported here, it is clear , no matter what I_MPI_EXTRA_FILESYSTEM's setup, there are problems with the writing out of U and V in fv3reg GSI , working on which is ongoing and would be updated here

[ShunLiu-NOAA]

@TingLei-NOAA your findings are similar to the streak feature in RRFS forecast. Let's have a short meeting and collect all information together.

[RussTreadon-NOAA]

Other instances of hafs_3denvar_glbens failures have found differences between loproc and updat in the T (see above) and delp fields (see here).

[TingLei-NOAA]

An update: After changing , for example, "call check( nf90_open(filenamein,nf90_write,gfile_loc,comm=mpi_comm_world,info=MPI_INFO_NULL) )" to "call check( nf90_open(filenamein,ior(nf90_write,nf90_mpiio),gfile_loc,comm=mpi_comm_world,info=MPI_INFO_NULL) )" in the update branch, In the regression test,
the control still show differences between loproc and hiproc in fv3_dynvars, while the update runs show identical files . However, for the spontaneous occurrences of this issue as @RussTreadon-NOAA found, I can't say this is just the solution. I will prepare a clean branch with all recent changes (including those in #694) for more systematic tests/verifications.