Run nesting and the new physics using the global-workflow #4

guoqing-noaa · 2024-04-20T22:22:37Z

This PR bases on the latest EMC develop branch, add the nesting functionality and the FV3_global_nest_v1 CCPP physics suite to the workflow.

A quick instruction to run a global nest experiment and/or use the new physics is given at:
https://docs.google.com/document/d/15IOLJ_GwoKcbjjbsX2ITgbSIJmIJB-m8mRulRn1ikYg/edit

…onfig.ufs

…tion

…lobal_nest

add the global_nest physics into the global-workflow

SamuelTrahanNOAA

In an effort to simplify my changes, you have created something that will be unmaintainable. There are some major problems here. You need to put back in some of my changes.

parm/config/gfs/config.ufs

SamuelTrahanNOAA · 2024-04-22T14:58:11Z

sorc/build_ufs.sh

+#MAKE_OPT+=" -DFASTER=ON"  #comment out this line for the global_nest runs
+


This should have been left in as a -f command line option. Users should not edit the build scripts and uncomment things.

-f command line option can be easily passed into build_ufs.sh but it is hard to be passed into build_all.sh
We will consider a full workflow instead of only the forecast part.
Users need to modify build_ufs.sh anyway to include the global_nest_v1 physics in the compiling time.

In that case, it is best to use -DFASTER=ON as the default. It was designed to be both faster and safer than the defaults.

I trust you. But I am not sure whether EMC will accept a default -DFASTER=ON. But we can try.

The best option is to have -f in build_all.sh and have it passed down. If you don't want to do that, we should have it be -DFASTER=ON by default. Then EMC will tell us how to make it optional in the review.

SamuelTrahanNOAA · 2024-04-22T15:01:50Z

ush/global_nest_patch.sh

+## patch link_workflow.sh
+sed -i -e 's#"hera").*#"hera")   FIX_DIR="/scratch2/BMC/wrfruc/Guoqing.Ge/fix" ;;#' -e 's#versions/fix.ver#versions/fix.nest.ver#' $HOMEgfs/sorc/link_workflow.sh


Several problems here:

Don't point to a user's personal directory.

Including a script to patch the scripts makes code management much harder. Use "if" statements or command-line arguments ("--nest") instead.

You should not replace the entire fix file hierarchy when all you are doing is adding a nested grid.

This is temporary for anyone interested to run this workflow. Otherwise, users will need to make those changes manually.
This file will not commit to the EMC repo.
The fix files will commit to the "authoritative“ location once they are accepted by EMC.

Yes, we can add a "--nest" option for link_workflow.sh

SamuelTrahanNOAA · 2024-04-22T15:02:58Z

ush/global_nest_patch.sh

+declare -rx HOMEgfs
+
+## patch config.base
+sed -i -e 's#export BASE_CPLIC=.*#export BASE_CPLIC="/scratch2/BMC/wrfruc/Guoqing.Ge/ufs-ar/ICS/global-nest"#' -e 's/export DO_NEST="NO"/export DO_NEST="YES"/' $HOMEgfs/parm/config/gfs/config.base


This should be set by the setup_workflow script via a --nest option. Including a script to patch the scripts makes code management much harder. Relying on commented-out code is a bad practice.

Get rid of it and put in my original changes to setup_workflow.

Do you mean we automatically set DO_NEST=YES and a new BASE_CPLIC location once we specify the "--nest" option?

I think whether to do nesting can be determined by a user through manually setting DO_NEST=YES in config.base under $EXPDIR

However, yes, we can make it happen automatically by the --nest option if needed.

I'd prefer to reduce the manual edits of config.* files as much as possible.

SamuelTrahanNOAA · 2024-04-22T15:08:31Z

ush/global_nest_patch.sh

+## patch build_ufs
+sed -i -e 's/FV3_GFS_v17_coupled_p8_ugwpv1"/FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1"/' -e 's/^#MAKE_OPT+=/MAKE_OPT+=/' $HOMEgfs/sorc/build_ufs.sh


Three problems here:

Relying on commented-out code is a bad practice.

There is no reason to keep FV3_global_nest_v1 out of the compiled CCPP_SUITES. Adding a suite has no negative effect (other than a slight increase in compile time).

Patching scripts makes code management more difficult.

The nested workflow does not need -DFASTER=ON, so it should not be hard-coded to that. Only the C768 needs it, mostly as a limitation of Hera's nodes. We should have the freedom to change the build options without having to edit them in two places.

Using a -f option in build_ufs.sh is simpler and better matches the current design.

This is a temporary file to avoid manual edits. It will not be committed to the authoritative repo.

Yes, as long as EMC accepts a new CCPP option in build_ufs.sh, we don't need this patch or user's edits.

The temporary fix should be the cleaner one. Leave the changes in build_ufs.sh.

Sure. I am fine with changing build_ufs.sh directly. I just stripped out all changes which will affect current GFS pre-v17 so as to make it ready for a PR to the EMC repo. But we can try your idea and modify if they have concerns. Thanks!

Having a clean set of changes is more important than avoiding future discourse with EMC. It'll be difficult to review when everything is hidden behind sed commands in a ush/global_nest_patch.sh script. You need to remove that sedding script and make the changes in each individual file instead. It'll be both easier to maintain and easier to review.

SamuelTrahanNOAA · 2024-04-22T15:09:48Z

versions/fix.nest.ver

+export am_ver=global-nest.20240419
+export orog_ver=global-nest.20240419
+export ugwd_ver=global-nest.20240419


The vast majority of these files are not grid-specific. They're climatology files on their own grid. Do we really need our own copy of everything?

This eliminates the need of "W384", "W768" etc. I think this simplifies the changes and make it easier to be accepted by EMC.
At the time I was working on this, I don't know what files are your new fix files.

We can either (1) make a new copy of every climatology file as the files size is not that big and acceptable;
or (2) create a new version but make those climatology files as links.
We can further discuss this in a tag up. Thanks!

I see no other way to get rid of the W grids, so I agree we'll have to have our own fix files unless EMC comes up with a better option.

Co-authored-by: Samuel Trahan (NOAA contractor) <39415369+SamuelTrahanNOAA@users.noreply.github.com>

guoqing-noaa · 2024-04-22T17:05:27Z

@SamuelTrahanNOAA Thanks for all the comments. I'm committing changes in this PR incrementally to the authoritative EMC repo and hope to complete it in the next few days.

SamuelTrahanNOAA · 2024-04-22T17:44:38Z

Your changes should not be merged to this repository while ush/global_nest_patch.sh exists. That is not a viable way to maintain the repository.

guoqing-noaa · 2024-04-22T17:47:54Z

Sam, I don't plan to merge this PR as this is a temporary solution. I intended to use this PR to facilitate discussions.
Also, FYI, this is a PR to the "nest" branch, not the "develop-AR" nor "develop-EMC" branch.

guoqing-noaa and others added 29 commits April 19, 2024 13:16

add parm/config/global-nest, links from ../gfs

d50754d

config.ufs, config.fcst: change from links to regular files

e3c85ff

update parameters for the global_nest_v1 physics in config.fcst and c…

a6ec298

…onfig.ufs

update ush/parsing_namelists_FV3.sh to include the FV3_global_nest op…

7e7fe2c

…tion

add parm/ufs/fv3/field_table_thompson_aero_tke_progsigma

c0208ef

use the new use the new field_table_thompson_aero_tke_progsigma for g…

c454e77

…lobal_nest

Merge pull request ufs-community#3 from guoqing-noaa/ccpp-global-nest

936e484

add the global_nest physics into the global-workflow

modify config.fcst to set DO_CA=NO

60d7d83

simplify the DO_CA change in config.fcst

f89540c

update config.stage_ic to include UFS-AR ICs

5d51776

set the MPI layout for the nested tile

f8af477

add the 'DO_NEST' variable in config.base

cc7dcd6

update config.ufs for DO_NEST

85b12e7

update exglobal_stage_ic.sh for DO_NEST

1d83e88

add one comment line about compiling with -DFASTER=ON in build_ufs.sh

e5315c9

update forecast_postdet.sh

91151af

update forecast_predet.sh

4be24e5

update parsing_namelists_FV3.sh

eb4cfe2

update parsing_model_configure_FV3.sh

023e510

add fix.nest.ver for DO_NEST

d207862

simlify fix.nest.ver

2bedcf3

add ush/global_nest_patch.sh

63447b3

minor bug fix in versions/fix.nest.ver

273a08a

bug fix on config.base

fd169d3

put changes to config/gfs/config.ufs

84c249c

merge changes to config/gfs/config.fcst

7b2f01f

remove global-nest

86de796

different parameter settings are now based on CCPP_SUITE not DO_NEST

409543b

update sorc/ufs_model.fd

bbc0e55

SamuelTrahanNOAA suggested changes Apr 22, 2024

View reviewed changes

Update parm/config/gfs/config.ufs

dd37f80

Co-authored-by: Samuel Trahan (NOAA contractor) <39415369+SamuelTrahanNOAA@users.noreply.github.com>

guoqing-noaa added 4 commits April 23, 2024 08:32

changes based on suggestions

469a63c

remove global_nest_patch.sh

53da372

change case prefix to GLOBAL-NEST

7c2dbaf

remove sorc/tmp.sh

623c196

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run nesting and the new physics using the global-workflow #4

Run nesting and the new physics using the global-workflow #4

guoqing-noaa commented Apr 20, 2024 •

edited

Loading

SamuelTrahanNOAA left a comment

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa Apr 22, 2024

SamuelTrahanNOAA Apr 22, 2024

guoqing-noaa commented Apr 22, 2024

SamuelTrahanNOAA commented Apr 22, 2024

guoqing-noaa commented Apr 22, 2024 •

edited

Loading

		#MAKE_OPT+=" -DFASTER=ON" #comment out this line for the global_nest runs

		## patch link_workflow.sh
		sed -i -e 's#"hera").*#"hera") FIX_DIR="/scratch2/BMC/wrfruc/Guoqing.Ge/fix" ;;#' -e 's#versions/fix.ver#versions/fix.nest.ver#' $HOMEgfs/sorc/link_workflow.sh

		## patch build_ufs
		sed -i -e 's/FV3_GFS_v17_coupled_p8_ugwpv1"/FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1"/' -e 's/^#MAKE_OPT+=/MAKE_OPT+=/' $HOMEgfs/sorc/build_ufs.sh

Run nesting and the new physics using the global-workflow #4

Are you sure you want to change the base?

Run nesting and the new physics using the global-workflow #4

Conversation

guoqing-noaa commented Apr 20, 2024 • edited Loading

SamuelTrahanNOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guoqing-noaa commented Apr 22, 2024

SamuelTrahanNOAA commented Apr 22, 2024

guoqing-noaa commented Apr 22, 2024 • edited Loading

guoqing-noaa commented Apr 20, 2024 •

edited

Loading

guoqing-noaa commented Apr 22, 2024 •

edited

Loading