feature/global_det_v1.0plus

[malloryprow]

Pull Request Testing

This PR includes a variety of changes and updates related to EVS v2.0:

1. Add "last" into dev drivers and ecf plots script names
2. Remove echo of $config in dev driver scripts
3. Change "past" to "last" in global_det wave image names
4. Remove use of LOOP_ORDER in METplus config files
5. Use subprocess.run and remove subprocess import when not used
6. Switch GFS and ECMWF order on headline plot so GFS is first
7. Update grid2grid and grid2obs stats job script formatting
8. Clean up logging messages
9. Mask out negative values prior to taking square root to avoid warning messages
10. Rename dev drivers scripts so they match ops ecf scripts
11. Add "set -x" to mail scripts
12. Define $NET before $HOMEevs
13. Remove PBS -V (Remove "PBS -V" from dev driver scripts #464)
14. Add 0.25 inch threshold for ETS global_det atmos headline plots
15. Update default MET version in plotting scripts for running standalone
16. Add $COMOUTosisaf
17. Fix y-axis scale on headline precip ETS plots per WPC request
18. Use PB2NC_SKIP_VALID_TIMES
19. Remove cat'ing of log files as LOG_TO_TERMINAL_ONLY = TRUE is now being use
20. Update long-term plots to show the ACC average of the period they display (all vs. 10 yr)
21. Remove ltop, uptrop, and trop grid2grid_pres_levs plots for Ozone
22. Make some warning messages debug or note statements.
23. Update drivers to use environment VDATE_END

Those is italics are in the Fix and Additions in EVS v2.0 document.

NOTE: Upon merging this PR, we will need to update the cron scripts for the renamed stats and plots drivers.

• Describe testing already performed for this Pull Request:
  
  I have been running the global_det prep, stats, and plots drivers in my crontab for the past two weeks. I compared the prep and stats output to the emc.vpppg parallel and the output for 20240706 all matched. I did not compare the plots tar files directly as I'm pointing to my stats and not the emc.vpppg parallel stats. I checked the logs and there are not unexpected warning or error messages.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

The testing for this PR is going to be a lot but we can take it chunk by chunk and work through it.

Set-up

1. Clone my fork and checkout branch feature/global_det_v1.0plus
2. ln -sf /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix fix
3. cd sorc; ./build.sh

✔️ global_det wave prep

4. cd dev/drivers/scripts/prep/global_det
5. In jevs_global_det_wave_prep.sh, set HOMEevs to the location of the clone. Run. (Let me know when you are going to run and I can tell you if you need to adjust INITDATE or not.)

✔️ global_det wave stats

6. cd dev/drivers/scripts/stats/global_det
7. In jevs_global_det_gfs_wave_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)

✔️ global_det wave plots

8. cd dev/drivers/scripts/plots/global_det
9. In jevs_global_det_wave_grid2obs_plots_last31days.sh and jevs_global_det_wave_grid2obs_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)

✔️ global_det atmos prep

10. cd dev/drivers/scripts/prep/global_det
11. In jevs_global_det_atmos_prep.sh, set HOMEevs to the location of the clone. Run. (Let me know when you are going to run and I can tell you if you need to adjust INITDATE or not.)

✔️ global_det atmos stats

13. cd dev/drivers/scripts/stats/global_det
14. In jevs_global_det_cfs_atmos_grid2grid_stats.sh and jevs_global_det_cfs_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
15. In jevs_global_det_cmc_atmos_grid2grid_stats.sh and jevs_global_det_cmc_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
16. In jevs_global_det_cmc_regional_atmos_grid2grid_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
17. In jevs_global_det_dwd_atmos_grid2grid_stats.sh, set HOMEevs to the location of the clone. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
18. In jevs_global_det_ecmwf_atmos_grid2grid_stats.sh and jevs_global_det_ecmwf_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
19. In jevs_global_det_fnmoc_atmos_grid2grid_stats.sh and jevs_global_det_fnmoc_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
20. In jevs_global_det_gfs_atmos_grid2grid_stats.sh and jevs_global_det_gfs_atmos_grid2obs_stats.sh,, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
21. In jevs_global_det_imd_atmos_grid2grid_stats.sh and jevs_global_det_imd_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
22. In jevs_global_det_jma_atmos_grid2grid_stats.sh and jevs_global_det_jma_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
23. In jevs_global_det_metfra_atmos_grid2grid_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
24. In jevs_global_det_ukmet_atmos_grid2grid_stats.sh and jevs_global_det_ukmet_atmos_grid2obs_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE or not.)
25. In jevs_global_det_atmos_long_term_stats.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run.

✔️ global_det atmos plots

26. cd dev/drivers/scripts/plots/global_det
27. In jevs_global_det_atmos_grid2grid_means_plots_last31days.sh and jevs_global_det_atmos_grid2grid_means_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
28. In jevs_global_det_atmos_grid2grid_precip_plots_last31days.sh and jevs_global_det_atmos_grid2grid_precip_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
29. In jevs_global_det_atmos_grid2grid_pres_levs_plots_last31days.sh and jevs_global_det_atmos_grid2grid_pres_levs_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
30. In jevs_global_det_atmos_grid2grid_sea_ice_plots_last31days.sh and jevs_global_det_atmos_grid2grid_sea_ice_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
31. In jevs_global_det_atmos_grid2grid_snow_plots_last31days.sh and jevs_global_det_atmos_grid2grid_snow_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
32. In jevs_global_det_atmos_grid2grid_sst_plots_last31days.sh and jevs_global_det_atmos_grid2grid_sst_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
33. In jevs_global_det_atmos_grid2obs_pres_levs_plots_last31days.sh and jevs_global_det_atmos_grid2obs_pres_levs_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
34. In jevs_global_det_atmos_grid2obs_ptype_plots_last31days.sh and jevs_global_det_atmos_grid2obs_ptype_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
35. In jevs_global_det_atmos_grid2obs_sfc_plots_last31days.sh and jevs_global_det_atmos_grid2obs_sfc_plots_last90days.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
36. In jevs_global_det_atmos_headline_plots.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Set envir to dev. Run. (Let me know when you are going to run and I can tell you if you need to adjust VDATE_END or not.)
37. In jevs_global_det_atmos_long_term_plots.sh, set HOMEevs to the location of the clone. Set COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2d. Run.

• Has the code been checked to ensure that no errors occur during the execution? Yes

• Do these updates/additions include sufficient testing updates? Yes

• Please complete this pull request review by 7/19/2024.
  

Pull Request Checklist

• Review the source issue metadata (required labels, projects, and milestone).

• Complete the PR description above.

• Ensure the PR title matches the feature branch name.

• Check the following:

• Instructions provided on how to run

• Developer's name is replaced by ${user} where necessary throughout the code

• Check that the ecf file has all the proper definitions of variables

• Check that the jobs file has all the proper settings of COMIN and COMOUT and other input variables

• Check to see that the output directory structure is followed

• Be sure that you are not using MET utilities outside the METplus wrapper structure

• After submitting the PR, select Development issue with the original issue number.

• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.

• Close the linked issue.

[PerryShafran-NOAA]

All stats jobs are underway.

[malloryprow]

👍 wave plots are good! Logs are good and final tar file size and image counts match emc.vpppg parallel.

Confirming that the image names have been changed from "pastXdays" to "lastXdays".

tar -tvf /lfs/h2/emc/ptmp/perry.shafran/evs/v2.0/plots/global_det/wave.20240708/evs.plots.global_det.wave.grid2obs.last90days.v20240708.tar
-rw-r--r-- perry.shafran/emc 125506 2024-07-09 15:38 evs.global_det.corr.htsgw_l0_ndbc_standard.last90days.fhrmean_valid00z_f384.latlon_0p25_glb.png
-rw-r--r-- perry.shafran/emc 138058 2024-07-09 15:38 evs.global_det.corr.htsgw_l0_ndbc_standard.last90days.fhrmean_valid00z_f384.latlon_0p25_gom.png
-rw-r--r-- perry.shafran/emc 124467 2024-07-09 15:38 evs.global_det.corr.htsgw_l0_ndbc_standard.last90days.fhrmean_valid00z_f384.latlon_0p25_hawaii.png

[PerryShafran-NOAA]

All the stats jobs have completed except for the gfs atmos grid2obs job, if you wish to take a look.

The long_term_stats job seems to be smaller than all the others, but perhaps it's supposed to be?

[malloryprow]

All jobs are done now!

The atmos stats jobs are good. 👍 There are ERROR and WARNING messages in the logs, but these are known and expected and will be resolved in beta5. The final stat files are all good.

For atmos plots, you'll want to run with VDATE_END=20240707 for every job but jevs_global_det_atmos_long_term_plots.sh. Additionally, in jevs_global_det_atmos_headline_plots.sh set envir=dev.

[PerryShafran-NOAA]

Awesome! Just to confirm, your thumbs up includes for long term stats?

I'll get the plot jobs started.

[malloryprow]

Yup! The long-term stats are set up a bit different than the "standard" stats jobs.

[PerryShafran-NOAA]

Plot jobs are all now underway.

[malloryprow]

I'll check them when I start working tomorrow. The GFS v17 T2O meeting is soon, and then I'm heading out after. Some of the plot jobs take an hour+ to run.

[PerryShafran-NOAA]

@malloryprow All the plot jobs are finished. I noted that I set the date for VDATE, not VDATE_END, and realized that those values are different because all the plot tarballs have a vdate of 20240708, and I set VDATE for 20240707. If I need to re-run anything, please let me know, but since you are checking tomorrow, those plot jobs can be compared.

[malloryprow]

I'd prefer if we can rerun with VDATE_END=20240708 and clear out the COMOUT. I'd like to compare the final tarball sizes with the emc.vpppg parallel.

[PerryShafran-NOAA]

OK will do!

[malloryprow]

Interesting! I see in the logs you ran with "-v VDATE_END=20240708", but the job ran with VDATE_END=20240709. I can see what happened in the driver scripts. export VDATE_END=$(finddate.sh $today d-1) overrode your passed environment variable. I can make the change for the code to check if VDATE_END exists in the environment and make $(finddate.sh $today d-1) the default.

[PerryShafran-NOAA]

Oh thanks for checking! Yeah, that would certainly be helpful here.

[malloryprow]

Just pushed the changes!

[PerryShafran-NOAA]

Thank you! I'll re-run once again.

When Samira's PR is merged, you'll also need to sync your fork. But you always pay attention to these things!

[malloryprow]

Will watch!

[PerryShafran-NOAA]

Re-runs underway. Hopefully this one is good!

[malloryprow]

🤞 May the third time be the charm

[malloryprow]

Woohoo!

mallory.row@dlogin06:~> ls -ltr /lfs/h2/emc/ptmp/perry.shafran/evs/v2.0/plots/global_det
total 8
drwxr-sr-x 2 perry.shafran emc 4096 Jul 10 12:47 headline.20240708
drwxr-sr-x 11 perry.shafran emc 4096 Jul 10 12:48 atmos.20240708

I'll watch the jobs and report back. I will watch for Samira's PR merge too.

[malloryprow]

The atmos headline plots is done running so I reviewed that. Log is good and headline images that run in prod match the emc.vpppg parallel. Confirming the change of GFS being listed before ECMWF in the long-term ACC plot.

[PerryShafran-NOAA]

All the plot jobs are finished - but for some reason accidentally neglected to submit the long term plot job. Just did so now. We should have a complete plot set when that finishes. Please check everything else.

[malloryprow]

Minus the long-term plots, the final plots tarballs are good. File sizes match the emc.vpppg parallel but the grid2grid_pres_levs which is expected because the Ozone lower troposphere, upper troposphere, and troposphere vertical profile plots were removed. Logs are good too.

I'll follow the long-term plots log.

[malloryprow]

The long-term plots jobs is done now. All looks well with the output and log.

All good with testing and reviewing the output.

[PerryShafran-NOAA]

Good news! Let me do the code review. I think you already took care of the ecf defs file, which is a pretty large chunk of the job, but let me double check.

[PerryShafran-NOAA]

Code works as expected. Approved for merge.