[develop] Add integration test job

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

This PR adds a test job to the workflow. It was originally written with pytest but because of some file naming issues, the python package unittest was used instead (see reasoning below). The test checks for the existence of netcdf files from the weather model. The hope is to expand the number of integration tests to eventually include regression testing.

The necessary scripts were added or modified to incorporate the integration job into the workflow. A wrapper script was also added.

unittest package reasoning:
In order to pass arguments into a pytest script a conftest.py file needs to reside in the testing script directory or the directory above. Since this is a scheduler job, the conftest.py file will need to reside in ufs-srweather-app/scripts, but that would conflict with the NCO naming requirement, and storing conftest.py a directory above wouldn't make much sense either. Switching to unittest became the next best option.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
• orion.intel
• hercules.intel
• cheyenne.intel
• cheyenne.gnu
• derecho.intel
• gaea.intel
• gaeac5.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
  ran fundamental on AWS: /contrib/Edward.Snyder/pytest/expt_dirs/fundamental
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

Updated docs where it referred to the workflow tasks for the SRW App.

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

[MichaelLueken]

@EdwardSnyder-NOAA -

Thank you for adding the integration test job and script into the SRW App! These changes look good and the integration tests successfully passed on Hercules:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE               8.06
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              11.05
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              29.18
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              17.48
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024022711  COMPLETE              25.93
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240227113  COMPLETE              51.82
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              13.74
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240227113243  COMPLETE              69.25
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202402271132  COMPLETE              28.52
MET_verification_only_vx_20240227113245                            COMPLETE               0.27
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240227113246               COMPLETE               7.80
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             263.10

Approving now.

[MichaelLueken]

@EdwardSnyder-NOAA -

The WE2E test, deactivate_tasks, doesn't actually run the weather model. This causes the test to hang, since it is still trying to run the integration tests without the weather model. It would probably be best to go into tests/WE2E/test_configs/wflow_features/config.deactivate_tasks.yaml and add:

metatask_integration_test:

to cancel out the integration test for this specific WE2E.

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

The WE2E test, deactivate_tasks, doesn't actually run the weather model. This causes the test to hang, since it is still trying to run the integration tests without the weather model. It would probably be best to go into tests/WE2E/test_configs/wflow_features/config.deactivate_tasks.yaml and add:

metatask_integration_test:

to cancel out the integration test for this specific WE2E.

Thanks for catching that! I made the change as requested.

Eddie

[RatkoVasic-NOAA]

Looks good to me. Tests passed on Hera:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE               9.27
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              12.49
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               8.89
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              15.74
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022  COMPLETE              28.77
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240227195  COMPLETE              13.70
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022719594  COMPLETE              20.43
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             109.29

[MichaelLueken]

All tests have successfully passed, except for two tests on Hera Intel - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 and pregen_grid_orog_sfc_climo. Both tests failed in run_fcst. Rocotorewind/rocotoboot have been used on these two tests. Once complete, I will add the updated summary and merge this PR.

[MichaelLueken]

Utilizing the Rocky8 nodes on Hera, the last two tests that needed to pass to move forward with this PR completed successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km_20240305060102                            COMPLETE              18.10
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024030  COMPLETE               7.64
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             755.64
get_from_HPSS_ics_HRRR_lbcs_RAP_20240305060107                     COMPLETE              15.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               6.55
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              14.89
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240305060111  COMPLETE              10.80
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               8.15
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202403  COMPLETE             232.28
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240305  COMPLETE             301.03
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202403050  COMPLETE             324.97
pregen_grid_orog_sfc_climo_20240305060118                          COMPLETE               7.09
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1702.41

Merging this PR now.