Release 1.4 update regression tests

[TomasTorsvik]

Create regression test suites aux_blom_noresm and aux_hamocc_noresm.

I propose to have a separate test suit for iHAMOCC to enable testing of different optional configurations for ocean BGC. For changes to the physics, I think in most cases it will be sufficient to test only the default iHAMOCC setup.
If you agree that this setup is useful, I will include this setup also for noresm2.1 and noresm2.3.

I have included a ERP test, testing for bit-reproducible results with different processor count and openmp enabled. This test currently fails at the COMPARE_base_rest stage. We have not included answer changing bug fixes in v1.4, so I'm not sure we will correct this for this version of the model. I wonder if it still makes sense to keep it in as a reminder not to rely on this setting, or is it better to take it out?

After merging this PR, I would like to create a new release v1.4.2 to be used in noresm2.0.8.

[TomasTorsvik]

I also tried to test with hybrid vertical coordinates enabled. However, I'm not sure this makes sense since the hybrid coordinate setup has changed significantly since v1.4.1. Should I just remove this option?

[mvertens]

@TomasTorsvik - In looking at the output of the ERP test for BLOM - it turns out that only 2 fields are different - plon and plat. That's because these two fields actually show differences in the masked values and that makes sense. Can you please verify that you are also just seeing these two differences. If so - I think we should keep this test for now - but work to remove these differences.

[TomasTorsvik]

@mvertens
I also see only 2 fields with differences, domi_aream and x2i_So_t. I don't see output for plon and plat, I wonder if output for theses fields were added by @jmaerz recently (PR #355).

(...)

 domi_aream   (domi_nx,domi_ny,time)  t_index =      1     1
 WARNING: Fill patterns differ between files
 FILLDIFF domi_aream                      
       3099   138240  (   232,   264,     1) (    46,    13,     1) (   103,   264,     1) (    76,     1,     1)
              107800   2.293334252825896E-04   0.000000000000000E+00 2.3E-04  2.293334252825791E-04 1.1E-01  1.000000000000000E+30
              114240   2.293334252825896E-04   0.000000000000000E+00          0.000000000000000E+00          8.977538875101008E-06
              138240  (   232,   264,     1) (   261,     1,     1)
          avg abs field values:    9.652188848852659E-05    rms diff: 1.5E-05   avg rel diff(npos):  1.1E-01
                                   9.497001911968092E-05                        avg decimal digits(ndif):  0.0 worst:  0.0
 RMS domi_aream                       1.5087E-05            NORMALIZED  1.5757E-01

(...)

x2i_So_t   (x2i_nx,x2i_ny,time)  t_index =      1     1
 WARNING: Fill patterns differ between files
 FILLDIFF x2i_So_t                        
       3099   138240  (   243,   142,     1) (    46,    13,     1) (    49,    17,     1) (    76,     1,     1)
              107800   3.037844577392381E+02   0.000000000000000E+00 2.7E+02  2.731500000000000E+02 1.1E-01  1.000000000000000E+30
              114240   3.037844577392381E+02   0.000000000000000E+00          0.000000000000000E+00          2.731500000000000E+02
              138240  (   243,   142,     1) (   261,     1,     1)
          avg abs field values:    2.745917192135553E+02    rms diff: 4.6E+01   avg rel diff(npos):  1.1E-01
                                   2.734403605441557E+02                        avg decimal digits(ndif):  0.0 worst:  0.0
 RMS x2i_So_t                         4.6313E+01            NORMALIZED  1.6902E-01

(...)

SUMMARY of cprnc:
 A total number of    321 fields were compared
          of which      2 had non-zero differences
               and    152 had differences in fill patterns
               and      0 had different dimension sizes
 A total number of      0 fields could not be analyzed
 A total number of      0 fields on file 1 were not found on file2.
  diff_test: the two files seem to be DIFFERENT 

[mvertens]

I was doing this in noresm2.5 and the file was an hbgcd file. In rerunning the latest test suite it seems that the ERP test is passing (in non-hybrid mode. (ERP_Ld3_P256.T62_tn14.NOINYOC.betzy_intel).
I'm trying to find the file that had the plon and plat differences - but I cannot find it.

[TomasTorsvik]

@mvertens
Could it be that the test failed right after PR #355 was merged? If so, it would make sense since the hbgcd output was changed in that PR.

[TomasTorsvik]

Regression test:
SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel and ERS_Ld3.T62_tn14.NOINYOC.betzy_intel pass with comparison to noresm2.0.7 with BLOM version v1.4.1.

[mvertens]

@TomasTorsvik - the test is passing now with #377. I've done so much testing today on the different components aux tests that I cannot back track to exactly what the failed test was. However #377 looks okay in terms of regression testing.

[TomasTorsvik]

@mvertens - great! I think we can assume that ERP is fine for noresm2.5. I will test for noresm2.1 and noresm2.3 as well (but that will not be on the release-1.4 branch).

[JorgSchwinger]

Looks good to me.