Add GFS v16.3.11[12] changes to develop

[RussTreadon-NOAA]

What new functionality do you need?

GSI PR #636 updated script exglobal_atmos_analysis.sh. This script is version controlled in global-workflow develop for NOAA-EMC/GSI develop.

This issue is opened to document the updating of exglobal_atmos_analysis.sh in g-w develop.

What are the requirements for the new functionality?

Update exglobal_atmos_analysis.sh in g-w develop to be consistent with GFS v16.3.11.

Acceptance Criteria

This issue will be complete when exglobal_atmos_analysis.sh in g-w develop is consistent with the changes introduced in GFS v16.3.11.

Tasks:

• Get CRTMv2.4.0.1 in spack-stack on R&Ds (need spack-stack/1.6.0)
• Update crtm_ver=2.4.0.1 in version files - Add new data sources used in GFS v16.3.12 #2283
• Update GSI hash to one that uses new CRTM version
• Update ufs-weather-model hash to one that uses new CRTM version
• Update UPP hash to one that uses new CRTM version
• Uncomment GMI line scripts/exglobal_atmos_analysis.sh (commented out in GFSv16.3.5[6] GSI updates #1404). Moved to issue GFSv16.3.15 - Update to obsproc.v1.2 #2291
• Add abi_g18 line back into scripts/exglobal_atmos_analysis.sh (removed in ops and in develop GFSv16.3.5[6] GSI updates #1404) - done in Changes to add NOAA-21 and GOES-18 radiances in monitor mode GSI#636 & in Add new data sources used in GFS v16.3.12 #2283
• Add lines for ATMS and CRIS into scripts/exglobal_atmos_analysis.sh (see comment below GFS.v16.3.11 - Enable the monitoring of NOAA-21 and GOES-18 radiances #1356 (comment)) - Add new data sources used in GFS v16.3.12 #2283
• Obtain updated GSI fix set to get (updates to global_convinfo.txt and global_satinfo.txt) - will be done in Add GFS v16.3.11 binary GSI fix file to EID locations #1919 New GSI fix set for GFS v16.3.12 #2282
• Update gsi_ver=20240208 in versions/fix.ver

[KateFriedman-NOAA]

Thanks for opening this issue and issue #1919 @RussTreadon-NOAA !

I previously made a list of items to take care of in develop in the ops issue (see the checklist items at the bottom of the main comment in issue #1356). How many of those will be covered by this issue? Are there any that I list in issue #1356 that need to be taken care of on a later date or not at all?

Also, I noticed today that this global_convinfo.txt if-block is in ops (dev/gfs.v16) but not yet in the develop version of config.anal. Should that also be added into config.anal in develop? Thanks!

[RussTreadon-NOAA]

GSI tag gfsda.v16.3.10 does not update global_convinfo.txt. No changes to global_convinfo.txt are listed in NOAA-EMC/GSI PR #636.

Issue #1919 only adds binary GSI fix file Rcov_crisn21 to the EIB gsi fix directory. Based on your comment I now see that issue #1919 is not complete.

GFS v16.3.10 also updates ASCII GSI fix files global_anavinfo.l127.txt and global_satinfo.txt. GSI issue #640 will update the GSI-fix submodule hash in GSI develop. This is not sufficient for how EID stages GSI fix files. When g-w clones NOAA-EMC/GSI it does not include the ASCII GSI-fix hash associated with the cloned snapshot of NOAA-EMC/GSI. g-w assumes the EID staged directory contains both ASCII and binary GSI fix files.

Issue #1919 needs to be expanded to include updates to global_anavinfo.l127.txt and global_satinfo.txt. This complicates #1919. We need to create a new GSI fix directory.

I can't comment on the DO-3 Spire if-block on lines 123 to 126 of dev/gfs.v16/parm/config/config.anal

[RussTreadon-NOAA]

Thanks for opening this issue and issue #1919 @RussTreadon-NOAA !

I previously made a list of items to take care of in develop in the ops issue (see the checklist items at the bottom of the main comment in issue #1356). How many of those will be covered by this issue? Are there any that I list in issue #1356 that need to be taken care of on a later date or not at all?

Also, I noticed today that this global_convinfo.txt if-block is in ops (dev/gfs.v16) but not yet in the develop version of config.anal. Should that also be added into config.anal in develop? Thanks!

Given the to-do list in #1356 does EID take care of synching ops GFS changes to dev? If so, g-w issues #1918 and #1919 which I created today are not needed and should be closed.

[KateFriedman-NOAA]

Given the to-do list in #1356 does EID take care of synching ops GFS changes to dev? If so, g-w issues #1918 and #1919 which I created today are not needed and should be closed.

@RussTreadon-NOAA Both issues are needed and I appreciate you opening them because they'd need to get opened soon anyway. I hadn't gotten to it yet so I'm glad they exist now. :)

Thanks for the feedback in your prior comment. I looked at NOAA-EMC/GSI#636 and updated the checklist in #1356 to note which ones were taken care of in the GSI PR and will subsequently be taken care of in this issue or #1919.

@RussTreadon-NOAA @ADCollard A few questions:

Update crtm_ver=2.4.0.1 in version files
Update GSI hash to one that uses new CRTM version
Update ufs-weather-model hash to one that uses new CRTM version
Update UPP hash to one that uses new CRTM version

1. I don't see anything related to CRTM in Changes to add NOAA-21 and GOES-18 radiances in monitor mode GSI#636. Do the updates in this issue need the newer CRTMv2.3.0.1? We need to get it installed on the R&Ds still.

Update obsproc version to 1.2 throughout version files

2. Does this issue need the new obsproc version used with it? I need to get obsproc.v1.2 installed on the R&Ds. I can start that sooner if it's needed in parallel with this update. Let me know, thanks!

Uncomment GMI line scripts/exglobal_atmos_analysis.sh (commented out in #1404).

3. @ADCollard when will GMI come back into the analysis for GFSv17? This is a checklist item in GFS.v16.3.11 - Enable the monitoring of NOAA-21 and GOES-18 radiances #1356.

4. @RussTreadon-NOAA will you be sending in a PR with the script changes? Wanna make sure we know who will be making which updates. Thanks!

[RussTreadon-NOAA]

Great questions @KateFriedman-NOAA.

To my knowledge of the several GSI repos in NOAA-EMC, only GSI-Monitor has an officially tasked code manager. I am unaware of assigned code managers for other GSI repos. I acted proactively today in support of @ADCollard. He has a many tasks (operations & JEDI) on his plate. Let's give him a chance to reply and we'll go from there.

[WalterKolczynski-NOAA]

Updated references to the version number since the annual CO2 update is being slotted in first.

[KateFriedman-NOAA]

@ADCollard @RussTreadon-NOAA Shall we wait until GFSv16.3.12 (active assimilation) update is in ops and then work to bring the v16.3.11 and v16.3.12 changes into global-workflow develop?

[ADCollard]

@KateFriedman-NOAA I would suggest waiting until then

[KateFriedman-NOAA]

I would suggest waiting until then

Works for me! I'll adjust this issue to cover both upgrades and pull the remaining checklist items from issue #1356 into this one so I can close that issue now that that version is in ops. Thanks!

[ADCollard]

This issue is addressed in global-workflow PR 2283.

Fix file are added in GSI-fix PR 13

[ADCollard]

@KateFriedman-NOAA Sorry for the slow followup to this. I went through the checklist and I had totally forgotten about needing to update the ufs-weather-model and the UPP! I assume I will need to ask @WenMeng-NOAA to provide the tag for UPP. Who should I approach for the ufs-weather-model?

[KateFriedman-NOAA]

@KateFriedman-NOAA Sorry for the slow followup to this. I went through the checklist and I had totally forgotten about needing to update the ufs-weather-model and the UPP! I assume I will need to ask @WenMeng-NOAA to provide the tag for UPP. Who should I approach for the ufs-weather-model?

@ADCollard No worries! For ufs-weather-model talk to @junwang-noaa. Thanks!

[ADCollard]

@WenMeng-NOAA @junwang-noaa Following on from the recent update to CRTM in operations in December, we need to merge these changes into the develop version of the global workflow. So we need to update the versions used in upp and the ufs-weather-model.

Can I just go ahead and make this change? Looking in ufs_model.fd there are a number of references to different versions of the CRTM. Should these all be consolidated to v2.4.0.1?

FV3/upp/jobs/JGLOBAL_ATMOS_NCEPPOST:export crtm_ver=${post_crtm_ver:-v2.2.6}
FV3/upp/jobs/J_NCEPPOST:export crtm_ver=${post_crtm_ver:-v2.2.6}
FV3/upp/jobs/J_NCEPPOST:#export FIXCRTM=${FIXCRTM:-${NWROOThps}/hwrf.${hwrf_ver}/fix/hwrf-crtm-2.0.6}
FV3/upp/ush/link_crtm_fix.sh:# 2. $FIXCRTM environment variable, or
FV3/upp/ci/spack.yaml:  - crtm@2.3.0
FV3/upp/scripts/run_upp:ln -fs $CRTMDIR/SpcCoeff/Big_Endian/imgr_g12.SpcCoeff.bin    ./
FV3/upp/scripts/run_upp:ln -fs $CRTMDIR/TauCoeff/ODPS/Big_Endian/imgr_g12.TauCoeff.bin    ./
FV3/upp/scripts/run_upp:ln -fs $CRTMDIR/SpcCoeff/Big_Endian/imgr_mt2.SpcCoeff.bin    ./
FV3/upp/scripts/run_upp:ln -fs $CRTMDIR/TauCoeff/ODPS/Big_Endian/imgr_mt2.TauCoeff.bin    
FV3/upp/scripts/exgfs_atmos_nceppost.sh:    # link satellite coefficients files, use hwrf version as ops crtm 2.0.5
FV3/upp/docs/Installation.rst:   crtm 2.3.0
FV3/upp/sorc/ncep_post.fd/CALRAD_WCLOUD_newcrtm.f:!> 2022-09-12 | Wen Meng       | Added cloud fraction changes for crtm/2.4.0
FV3/upp/modulefiles/wcoss2.lua:crtm_ver=os.getenv("crtm_ver") or "2.4.0"
FV3/upp/modulefiles/wcoss2.lua:load(pathJoin("crtm", crtm_ver))
FV3/upp/modulefiles/upp_common.lua:  {["crtm"]            = "2.4.0"},
FV3/upp/modulefiles/s4:module load crtm/2.3.0
FV3/upp/modulefiles/cheyenne_gnu:module load crtm/2.3.0
FV3/upp/modulefiles/cheyenne:module load crtm/2.3.0
FV3/upp/modulefiles/jet:module load crtm/2.3.0
FV3/upp/modulefiles/gaea.lua:  {["crtm"]        = "2.4.0"},
FV3/upp/modulefiles/gaea-c5.lua:  {["crtm"]            = "2.4.0"},
FV3/upp/modulefiles/wcoss2_a.lua:crtm_ver=os.getenv("crtm_ver") or "2.4.0"
modulefiles/ufs_common.lua:  {["crtm"]            = "2.4.0"},
modulefiles/ufs_stampede.intel:module load crtm/2.3.0
modulefiles/ufs_wcoss2.intel.lua:  {["crtm"]            = "2.4.0"},
tests/ufs_common.lua:  {["crtm"]            = "2.4.0"},

[ADCollard]

The GSI update is addressed in GSI PR #684

[junwang-noaa]

@ADCollard Since upp is the component with dependency on crtm. I'd suggest to update upp with crtm 2.4.1 first. Also may I ask how you test crtm 2.4.1 in GSI. I think those components are using spack-stack, the same as the ufs-weather-model. So far in the latest spack-stack version 1.6.0, I see crtm is still 2.4.0.1. We may need to ask the library team to install the crtm 2.4.1 in the spack-stack package so that ufs weather model, GSI and upp can use it.

spack-stack 1.6.0 library updates are listed here:

JCSDA/spack-stack#924 (comment)

@AlexanderRichert-NOAA @climbfuji FYI.

[WenMeng-NOAA]

@ADCollard @KateFriedman-NOAA @junwang-noaa The UPP develop branch was updated with crtm/2.4.0.1 for most of R&D platforms. I will update WCOSS2 and WCOSS2 Acorn soon. After that, an upp submodule revision update PR will be submitted to ufs-weather-model.

[WenMeng-NOAA]

@KateFriedman-NOAA @ADCollard @junwang-noaa The UPP develop branch was committed at d597055 for upgrading crtm to 2.4.0.1 on WCOSS2 and Acorn. The crtm upgrade in modulefiles for R&D platforms like Hera and Orion was done early in another PR.

[KateFriedman-NOAA]

Thanks @WenMeng-NOAA !

@junwang-noaa @ADCollard Checking on this, do we have an updated UWM with crtm/2.4.0.1 and the updated UPP yet? Thanks!

[ADCollard]

@KateFriedman-NOAA @junwang-noaa

I am not entirely sure how to proceed. The move to 2.4.0.1 should produce identical results for both UWM and UPP, so I would think that it would be straightforward to update. Should I just try to do it myself?

I assume the UPP is ready with the branch committed at d597055.

I am assuming that final testing should wait for any fixes related to the Rocky 8 transition?

[KateFriedman-NOAA]

@ADCollard So, we're temporarily adding a UPP submodule to global-workflow develop as we introduce Hera Rocky8 changes (in final review/testing right now). The UPP hash we're using includes the update to crtm/2.4.0.1. So for now, we just need the crtm version update in a new hash of ufs-weather-model.

Will let @junwang-noaa comment on where the crtm update in ufs-weather-model stands.

[ADCollard]

@KateFriedman-NOAA @junwang-noaa @WenMeng-NOAA Just following up on this again. As I understand it the global workflow develop version is using CRTM v2.4.0.1 for GSI and UPP. So this issue is effectively solved.

The ufs-weather-model still has references to earlier CRTM versions, but as far as I can tell CRTM is not used there except through the UPP (which, of course, has its own repository).

So is there any further action required to address this issue?

[CatherineThomas-NOAA]

@KateFriedman-NOAA @junwang-noaa @WenMeng-NOAA
Pinging on this again as we are now having failures in the GSI since this hasn't been merged to the workflow yet but the GSI fix files were updated.

[KateFriedman-NOAA]

Yes, let's try to get this in quickly now. The crtm is now 2.4.0.1 in develop as well so that should override the components if they are behind (UWM).

@ADCollard Your branch for this PR is behind current develop by a lot now. Please sync it with develop and then we can test and try to wrap this up.

Has anyone tested these changes on top of current develop yet? Wanna make sure there aren't more changes needed or new issues.

[KateFriedman-NOAA]

FYI, related to this, the obsproc.v1.2 update is finally going into ops soon. NCO just let us know that it'll be bundled in the GFSv16.3.15 upgrade that was just handed off. I'll process that version update after it goes into ops.

[ADCollard]

@KateFriedman-NOAA I am doing that now. I built the develop version of global-workflow yesterday and noted that the GSI was built with CRTM-2.4.0.1, so I assumed that all was well. Although the ufo-weather-model still uses older (and inconsistent) CRTM versions.

[ADCollard]

@KateFriedman-NOAA @junwang-noaa @WenMeng-NOAA Pinging on this again as we are now having failures in the GSI since this hasn't been merged to the workflow yet but the GSI fix files were updated.

@CatherineThomas-NOAA What is the issue with the GSI right now? Is it related to HIRS?

[CatherineThomas-NOAA]

@ADCollard
The GSI fails in the correlated obs mod:

48: [048]correlated_obsmod*upd_varch_(): >>> ERROR <<<  improperly set GSI_BundleErrorCov

CrIS N21 is now in the anavinfo file under the correlated error section, but the exglobal script doesn't have that sensor in the OBS_INPUT yet.

[KateFriedman-NOAA]

@KateFriedman-NOAA I am doing that now. I built the develop version of global-workflow yesterday and noted that the GSI was built with CRTM-2.4.0.1, so I assumed that all was well. Although the ufo-weather-model still uses older (and inconsistent) CRTM versions.

@junwang-noaa Is the newer crtm/2.4.0.1 within any PRs headed towards UWM develop? Or, do you forsee any issues if we set crtm_ver=2.4.0.1 at the workflow level but UWM is the older version?

@ADCollard Let me know if I can assist with testing or anything.

[ADCollard]

@KateFriedman-NOAA @CatherineThomas-NOAA I have synced my fork with develop and it built successfully. I will test now but if you want to test it the build is at
/scratch1/NCEPDEV/da/Andrew.Collard/git/global-workflow_ADC

[ADCollard]

@KateFriedman-NOAA @CatherineThomas-NOAA I ran my fork through a full cycle using the c96 test data on Hera. There were no failures.

ufs appears to compile with /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.1/envs/unified-env-rocky8/install/intel/2021.5.0/crtm-2.4.0-alizlgl but runs with crtm-fix/2.4.0.1_emc loaded. Apparently without incident.

[KateFriedman-NOAA]

Wonderful! @ADCollard If you're ready and @junwang-noaa has no concerns, we can start CI testing on your PR to make sure all is well for the various GFS modes. Our CI test system doesn't yet test on WCOSS2...has this been run there?

Please bring your PR out of draft and I can kick off CI testing. Thanks!

[ADCollard]

@KateFriedman-NOAA It is now out of draft. I will run the test on WCOSS today.

[KateFriedman-NOAA]

@ADCollard Great, thanks! Assuming the WCOSS2 test goes well I'll fire off the CI tests on Hera.

[ADCollard]

@KateFriedman-NOAA The global workflow test case ran for two full cycles on WCOSS2 without failures.

[KateFriedman-NOAA]

@KateFriedman-NOAA The global workflow test case ran for two full cycles on WCOSS2 without failures.

Excellent...I'll fire off our CI tests on Hera and Orion. Assuming those run well we can merge this on Monday most likely.

[KateFriedman-NOAA]

PR #2283 has been merged into develop. Closing this as complete.