Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build UPP standalone executable via cmake #249

Closed
WenMeng-NOAA opened this issue Jan 4, 2021 · 6 comments · Fixed by #257
Closed

Build UPP standalone executable via cmake #249

WenMeng-NOAA opened this issue Jan 4, 2021 · 6 comments · Fixed by #257
Labels
enhancement New feature or request

Comments

@WenMeng-NOAA
Copy link
Collaborator

Set up a new function of building UPP standalone executable via cmake instead of GNC makefile.

@WenMeng-NOAA WenMeng-NOAA added the enhancement New feature or request label Jan 4, 2021
@fossell
Copy link
Contributor

fossell commented Jan 7, 2021

@WenMeng-NOAA - Do you want our assistance since we do this for community and UFS apps? We should at least make sure our procedures that we distribute are the same. @mkavulich @kayeekayee

@WenMeng-NOAA
Copy link
Collaborator Author

@fossell I got a sample of cmake build with the NCEPLIBS from the hpc-stack from EMC/EIB colleagues. I started on setting up script and modulefiles for WCOSS, Hera and Orion in my branch. We definitely need assistances from the DTC team. I will contact with @mkavulich and @kayeekayee when my preliminary version is done. Thanks!

@WenMeng-NOAA
Copy link
Collaborator Author

@fossell @kayeekayee I just proposed a plan of unifying name convention to Arun and Huiya:

  1. unify name convention in UPP as:
    executable: upp.x (now ncep_post)
    library: upp.a (already implemented)
  2. Add new UPP standalone cmake build function for building executable and library (upp.a, include/*mod)
    The cmake build will use the hpc-stack libraries on WOSS, Hera, Orion, Jet
    ** keep current gmake build function a while for UPP user transition
  3. Change UPP github repository name EMC_post to UPP(?? other names)

Please let me know the comments/suggestions/concerns from the DTC.

Thanks,

@SMoorthi-emc
Copy link
Contributor

SMoorthi-emc commented Jan 7, 2021 via email

@WenMeng-NOAA
Copy link
Collaborator Author

WenMeng-NOAA commented Jan 7, 2021 via email

@SMoorthi-emc
Copy link
Contributor

SMoorthi-emc commented Jan 7, 2021 via email

@WenMeng-NOAA WenMeng-NOAA linked a pull request Jan 15, 2021 that will close this issue
EricJames-NOAA pushed a commit to EricJames-NOAA/UPP that referenced this issue Dec 14, 2022
* adding files for getting nomads data
new files:
         ush/NOMADS_get_extrn_mdl_files_grib.sh
         ush/NOMADS_get_extrn_mdl_files_nemsio.sh

* updated code for getting data online in one file
new file:
     ush/NOMADS_get_extrn_mdl_files.sh
deleted files:
     ush/NOMADS_get_extrn_mdl_files_grib.sh
     ush/NOMADS_get_extrn_mdl_files_nemsio.sh

* Enable ensemble forecasts (NOAA-EMC#245)

* Modify workflow to enable ensemble forecasts.

Summary of modifications:
------------------------
* Introduce the new workflow variables DO_ENSEMBLE and NUM_ENS_MEMBERS.  The user can enable ensemble forecasts by setting DO_ENSEMBLE to "TRUE" and NUM_ENS_MEMBERS to the number of ensemble members to use.
* When running ensemble forecasts, create/insert a set of ensemble member directories and create the cycle directories under these member directories.  These ensemble member directories are placed at the directory level that cycle directory levels would be placed when not running ensemble forecasts.
* Regardless of whether or not ensembles are enabled, change location where external model files are staged so that they are not in the cycle directories but instead one (without ensembles) or two (with ensembles) directory levels up.  In the case with ensembles, this needs to be done so that the external model files are not duplicated within each ensemble member directory; they do not need to be because all ensemble members use the same external model files.  This is also done for the case without ensembles in order to minimize the difference in worklfow behavior between the with and without ensemble cases.  To make this change of location of external model files, the new workflow variable EXTRN_MDL_FILES_BASEDIR is introduced (it is not a user-specified variable but a secondary one).
* Add two new WE2E tests for running ensemble forecasts, one in community mode (community_ensemble) and another in NCO mode (nco_ensemble).

Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Fix/add/delete comments and/or informational and/or error messages.
(B) Remove commented out code.
(C) Change location where external model files are staged so that they are not in the cycle directories (which are now underneath each ensemble member directory) but instead one level up.  This needs to be done so that the external model files are not duplicated within each ensemble member directory; they do not need to be because all ensemble members use the same external model files.
(D) Add a call to the new function set_FV3nml_stoch_params() that takes a base FV3 namelist file and generates from it a new FV3 namelist file for each ensemble member containing a unique set of stochastic parameters (relative to other ensemble members) and places it at the top level of that ensemble member's directory (so that all cycles in that member directory can create symlinks to it).
(E) Rename the variable FV3_NML_BASE_FN to FV3_NML_BASE_SUITE_FN to clarify that it specifies the name of the FV3 namelist file for the base physics suite (which is used to generate the namelist file specific to the user-specified physic suite).  This is done to better distinguish this base namelist file from the base namelist file used to generate namelist files for the various ensemble members.  (The name of the latter is specified in the new workflow variable FV3_NML_BASE_ENS_FN.)
(F) Introduce the new workflow variable FV3_NML_BASE_ENS_FN that specifies the name to use for the base FV3 namelist file from which to generate the namelist file for each ensemble member.  This variable is not used if not running ensemble forecasts (i.e. if DO_ENSEMBLE is not set to "TRUE").
(G) Add the local variable dummy_cyc_dir that specifies the (dummy) directory with respect to which to set the relaive paths of the fixed files (i.e. those in the FIXam directory) in the FV3 namelist file.  When running ensembles, this path is two levels up from the cycle directory; without ensembles, it is only one level up (as was originally the case).

File-by-file description of modifications:
-----------------------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES: (C)

jobs/JREGIONAL_RUN_FCST:
* Change CYCLE_DIR to cycle_dir since it is a local variable in this context (it is an argument to the script exregional_run_fcst.sh).

jobs/JREGIONAL_RUN_POST:
* For NCO mode, change directory in which output from the RUN_POST_TN task is stored such that if running ensemble forecasts, subdirectories are created under COMOUT_BASEDIR for each ensemble member.  This is done via the variable SLASH_ENSMEM_DIR, which is set to either "/mem$NN" where $NN is the member number (if running ensemble forecasts) or to a null string (if not running ensemble forecasts).  For community mode, the output from the post task is under CYCLE_DIR, which now gets set in the rocoto XML such that it is under an ensemble member directory (see below description of modifications to ush/templates/FV3SAR_wflow.xml).

modulefiles/tasks/hera/make_ics.local:
* Add wgrib2 (must have been removed by mistake?).

modulefiles/tasks/hera/make_lbcs.local:
* Add wgrib2 (must have been removed by mistake?).

scripts/exregional_make_grid.sh: (A), (D)

scripts/exregional_make_ics.sh: (C)

scripts/exregional_make_lbcs.sh: (C)

scripts/exregional_run_fcst.sh:
* Change CYCLE_DIR to cycle_dir since it is a local variable.
* Add a check such that if running ensemble forecasts, the symlink for the FV3 namelist file that must be present in the cycle directory points to the namelist file at the top level of the ensemble directory under which that cycle directory is located.

tests/baseline_configs/config.community_ensemble.sh:
* New workflow configuration file to perform WE2E test of ensemble forecasts in community mode.

tests/baseline_configs/config.nco_ensemble.sh:
* New workflow configuration file to perform WE2E test of ensemble forecasts in NCO mode.

tests/baselines_list.txt:
* Add two new WE2E tests for running ensemble forecasts, one in community mode (community_ensemble) and another in NCO mode (nco_ensemble).

ush/config_defaults.sh: (A), (E), (F)
* Introduce the new workflow variable DO_ENSEMBLE that specifies whether or not to run ensemble forecasts.  Enable ensemble forecasts by setting DO_ENSEMBLE to "TRUE".
* Introduce the new workflow variable NUM_ENS_MEMBERS that specifies the number of ensemble members.  This variable is not used if DO_ENSEMBLE is not set to "TRUE".

ush/generate_FV3SAR_wflow.sh: (A), (B), (D), (E), (G)
* Add new ensemble-related parameters to the "settings" variable that is used to customize the jinja2 template for the rocoto XML file.  These new parameters allow the resulting XML to loop over ensemble members, rename rocoto tasks and log files such that they contain the member number (and are thus unique), and modify cycle directories so that they are member-specific.

ush/set_FV3nml_sfc_climo_filenames.sh: (G)

ush/set_FV3nml_stoch_params.sh
* File to define new function that takes a base FV3 namelist file and generates from it a new FV3 namelist file for each ensemble member containing a unique set of stochastic parameters (relative to other ensemble members) and places it at the top level of that ensemble member's directory (so that all cycles in that member directory can create symlinks to it).

ush/setup.sh: (E), (F)
* Rename FCST_LEN_HRS_MAX to fcst_len_hrs_max since it is a local variable.
* Introduce the new workflow variable EXTRN_MDL_FILES_BASEDIR that specifies the base directory under which the external model files will be staged.  Under this directory, a subdirectory will be created for each external model (one for ICs, another for LBCs if different from the one for ICs), and under these, subdirectories will be created for each cycle in which to stage the files.  Note that EXTRN_MDL_FILES_BASEDIR is a secondary variable in the sense that it is not user-specifiable.
* Rename FV3_NML_BASE_FP to FV3_NML_BASE_SUITE_FP for the same reason as renaming of FV3_NML_BASE_FN to FV3_NML_BASE_SUITE_FN (see (E) above).
* Create new workflow variable FV3_NML_BASE_ENS_FP that specifies the full path to the base FV3 namelist file from which the namelist files for the individual ensemble members are generated.
* Introduce the new workflow array variable ENS_MEMBER_DIRS.  If running ensemble forecasts, set its elements to the ensemble member directories immediately under the experiment directory.
* If running ensemble forecasts, create the ensemble directories specified in the new workflow array variable ENS_MEMBER_DIRS.
* Record new variables to the workflow variable definitions file.

ush/templates/FV3SAR_wflow.xml:
* Bug fix - Change file name "make_grid_task_complete.txt" to "&MAKE_GRID_TN;_task_complete.txt" to make it changeable with the task name.
* Remove CYCLE_BASEDIR as an environment variable from the GET_EXTRN_ICS_TN and GET_EXTRN_LBCS_TN tasks.  This variable is no longer needed because the external model files are now staged outside of the cycle directories (under EXTRN_MDL_FILES_BASEDIR).
* Place a jinja2-controlled metatask around all tasks starting with MAKE_ICS_TN that loops over all ensemble members if do_ensemble is set to TRUE.
* For tasks that are within the metatask that loops over the ensemble members, add a string (uscore_ensmem_name) that identifies the ensemble member to task names, corresponding job names, and log file names.  Note that this variable is set to an empty string if not running ensemble forecasts.
* For tasks that are within the metatask that loops over the ensemble members, add a string (slash_ensmem_dir) that inserts the ensemble member directory to the definition of CYCLE_DIR (since when running ensembles, the cycle directories are under the member directories).  Note that this variable is set to an empty string if not running ensemble forecasts.
* Set the ensemble index (ENSMEM_INDX) as an environment variable in the RUN_FCST_TN task.  This is needed in the ex-script exregional_run_fcst.sh to be able to the symlink in the cycle directory to the FV3 namelist file in the correct ensemble member directory.  Note that this variable is set to an empty string if not running ensemble forecasts (in that case, it is not used).
* Set the ensemble member subdirectory preceded by a slash (SLASH_ENSMEM_DIR) as an environment variable in the RUN_POST_TN task.  This is needed in NCO mode when setting the directory in which to place the output of UPP.  Note that this variable is set to an empty string if not running ensemble forecasts.

ush/valid_param_vals.sh:
* Specify valid values for the new workflow variable DO_ENSEMBLE.

* Minor changes to code comments.

* Apparently there is now a requirement in the FV3 code that consv_te be set to 0 on any regional grid.  Make this change for the FV3_CPT_v0 suite (which is the only one for which consv_te had been set to a nonzero value).

* Bug fix in a diag_table that should already be in the develop branch.

* Bug fix -- Fix inconsistency in the way the ensemble member directories are named in different scripts.

The workflow generation scripts create ensemble directories named, e.g., mem1, mem2, ..., mem8, but the exregional_run_fcst.sh script assumes they are mem01, mem02, ..., mem08.  Make these consistent.  Now, the naming convention used depends on whether or not leading zeros are included in NUM_ENS_MEMBERS.  For example, if NUM_ENS_MEMBERS is set to "8", then the member directory names will be mem1, mem2, ..., mem8; and if NUM_ENS_MEMBERS is set to "08", then the member directory names will be mem01, mem02, ..., mem08.

* Add new WE2E test to test use of leading zeros in ensemble member names.

* Change directory structure so that ensemble member directories are beneath the cycle directories (instead of the opposite).  Details below.

Summary of modifications:
------------------------
* Place cycle directories above ensemble member directories, i.e. each cycle directory will contain a full set of ensemble member subdirectories that are used as the run directories.  Previously, it was the other way around, i.e. each member directory contained all cycle subdirectories.
* Move the external model directories into each cycle directory (instead of being in their own directory called extrn_mdl_files under the main experiment directory).
* During the experiment generation step, generate the full list of cycle dates/times to run and create a directory for each cycle.
* Add capability to have more than one cycle.  This capability was previously present but was inadvertently disabled during transition to generating the rocoto XML using a jinja2 template.
* In order to test the capability of the workflow to run multiple cycles (possibly on different days), modify the WE2E tests community_ensemble_2mems and nco_ensemble so that there are two cycle hours per day.  Also, modify community_ensemble_2mems so that the starting and ending days are different (one day later).

Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Change the location where external model files are stored to be under each cycle directory (instead of a separate directory specified by EXTRN_MDL_FILES_BASEDIR under the main experiment directory).
(B) Insert the new environment variable SLASH_ENSMEM_SUBDIR anywhere CYCLE_DIR appears.  This variable is passed in by the rocoto XML.  If not running ensemble forecasts, it is simply set to an empty string, and if running ensembles, it is set to the string "/${ensmem_subdir}" where ensmem_subdir is the subdirectory of the current ensemble member under the current cycle directory.  This allows the subdirectories containing ICS, LBCS, and RESTART files to be placed directly under the current cycle directory when NOT running ensembles and for them to be placed under the current ensemble member directory (which is one level down from the current cycle directory) when running ensembles.
(C) For clarity, add new local variable run_dir that gets set to the run directory based on the current cycle and, if applicable, the ensemble member.
(D) Call the new function create_diag_table_files (in the new file create_diag_table_files.sh) to create diagnostics table files.
(D) For correctness, rename the local variable dummy_cyc_dir to dummy_run_dir.
(E) Remove any use of EXTRN_MDL_FILES_BASEDIR since it is no longer needed as a workflow variable.
(F) Remove any use of ENS_MEMBER_DIRS since it is no longer needed as a workflow variable.
(V) Remove unused code.
(W) Edit informational and/or error messages.
(X) Remove trailing whitespace.
(Y) Remove commented out code.
(Z) Edit comments.

File-by-file description of modifications:
-----------------------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES: (A)

jobs/JREGIONAL_MAKE_ICS: (B)

jobs/JREGIONAL_MAKE_LBCS: (B)

jobs/JREGIONAL_RUN_FCST: (B), (C), (Z)
* Pass in ENSMEM_INDX and SLASH_ENSMEM_SUBDIR as arguments to the function exregional_run_fcst().

jobs/JREGIONAL_RUN_POST: (B), (Z)
* Create the new local variable run_dir in which to store the path to the run directory (for the current cycle and possibly ensemble member).
* In NCO mode, change location where ensemble directories are created to be under the cycle directory instead of above it (analogous change to NCO mode as is done in (B) for community mode).
* In community mode, place the postprd subdirectory under the run directory instead of under CYCLE_DIR (since now, cycle directories are one level up if running ensembles and thus would be the incorrect place to create postprd).
* Create the new argument cdate to the function exregional_run_post() and pass in the environment variable CDATE for its value (this is instead of using CDATE directly in exregional_run_post()).
* Change the argument cycle_dir of the function exregional_run_post to run_dir since that's what we really want in that function.  This is instead of passing in cycle_dir and then forming run_dir.

scripts/exregional_make_grid.sh: (D)

scripts/exregional_make_ics.sh: (A), (X)

scripts/exregional_make_lbcs.sh: (A), (X)

scripts/exregional_run_fcst.sh: (C), (W)
* Introduce new input arguments ensmem_indx and slash_ensmem_subdir that get set to the rocoto-specified environment variables ENSMEM_INDX and SLASH_ENSMEM_SUBDIR, respectively, in the call to this function in jobs/JREGIONAL_RUN_FCST.
* Change cycle_dir to run_dir in most places to make the directory name more general.  This is because the run directory will be the cycle directory only when not running ensembles.  When running ensembles, the run directory will be one of the ensemble member directories, which will be one level down from the current cycle directory.
* Fix typo where there is an extra "}" printed after $target in error messages.
* Use the new workflow array variable FV3_NML_ENSMEM_FPS (which contains the full paths to the FV3 namelist files for each ensemble member) when creating a link in the run directory to the FV3 namelist file for the current ensemble member.  Note that these namelist files are cycle-independent and thus are created only once during the experiment generation step.
* Move the creation of diagnostics table files to a new function (in ush/create_diag_table_files.sh), and call that function during experiment generation (in ush/generate_FV3SAR_wflow.sh) instead of here in exregional_run_fcst.sh.  We do this because the diagnostics table files depend only on the cycle, not the ensemble member.  Thus, since we know the cycles to run at experiment generation time, we generate the diagnostics file for each cycle then and place each in its corresponding cycle directory.
* If running ensembles, create symlinks in the run directory to the diagnostics table and model configure files in the cycle directory (which will be one level up from the run directory).  We don't do this when NOT running ensembles because in that case, the run directory is the cycle directory (and these two files already exist in that directory; they are created during experiment generation time).

scripts/exregional_run_post.sh: (C), (Y)
* Create the new input argument cdate (which gets set to the global variable CDATE in the call to this function (exregional_run_post)) and use it instead of the global variable CDATE.
* Change the argument cycle_dir to run_dir since that's more useful in this function.  This is instead of passing in cycle_dir and then forming run_dir.
* Make the local variables "POST_..." lowercase to follow the convention that local variables be in lower case.

tests/baseline_configs/config.community_ensemble_2mems.sh:
* Modify settings in this test configuration so that the starting and ending days of the cycles are not the same and so that there are two cycle hours per day.  This is to have more thorough testing of the ensembles feature in community mode.

tests/baseline_configs/config.nco_ensemble.sh:
* Modify settings in this test configuration so that there are two cycle hours per day.  This is to have more thorough testing of the ensembles feature in NCO mode.

ush/create_diag_table_files.sh:
* New file containing a function that creates a diagnostics table file for each cycle date and places it in the corresponding cycle directory.

ush/create_model_config_files.sh:
* New file containing a function that creates a model configuration file for each cycle date and places it in the corresponding cycle directory.

ush/set_cycle_dates.sh:
* New function that sets all the cycle dates/times to run.

ush/generate_FV3SAR_wflow.sh: (D)
* For clarity and consistency with other scripts, change variable name from slash_ensmem_dir to slash_ensmem_subdir.
* Change "settings" variable used to set parameters in the jinja template for the rocoto XML to add capability to have more than one cycle.  This capability was previously present but was inadvertently disabled during transition to generating the rocoto XML using a jinja2 template.
* Use the new workflow array variable ALL_CDATES (containing all the cycle dates/times to run) to create all the cycle directories.  Previously, the cycle directories were created during the make_ics or make_lbcs task, but it is clearer to do it during experiment generation.  Also, it now must be done during experiment generation because now, the model configuration file(s) and possibly also the diagnostics table file(s) (if the MAKE_GRID_TN step is being skipped), which are cycle-dependent but ensemble-member-independent, are created and placed in the cycle directories during experiment generation.
* Call the new function create_model_config_files() to create a model configuration file within each cycle directory.
* If not running the MAKE_GRID_TN task, call the new function create_diag_table_files() to create a diagnostics table file within each cycle directory.

ush/set_FV3nml_sfc_climo_filenames.sh: (D)

ush/set_FV3nml_stoch_params.sh: (Z)
* For consistency with other scripts, rename the variable fv3_nml_ens_fp to fv3_nml_ensmem_fp.
* Use the new workflow array variable FV3_NML_ENSMEM_FPS (which contains the full paths to the FV3 namelist files for each ensemble member) to set the full path to the current ensemble member's FV3 namelist file.

ush/setup.sh: (E), (F), (V), (Z)
* Call the new function set_cycle_dates() to set the new worklow array variable ALL_CDATES containing all the cycle days/times to be run.
* Set the new workflow variable NUM_CYCLES to the number of elements in ALL_CDATES.
* Set the new workflow array variable ENSMEM_NAMES containing the names of the ensemble members.
* Set the new workflow array variable FV3_NML_ENSMEM_FPS containing the full paths to the FV3 namelist files of the ensemble members.
* Remove creation of ensemble member directories.  These are now created in the j-jobs of the MAKE_ICS_TN or the MAKE_LBCS_TN task (whichever runs first).

ush/templates/FV3SAR_wflow.xml: (Y)
* Modify jinja code to allow for multiple cycles to be run.  This capability was previously present but was inadvertently disabled during transition to generating the rocoto XML using a jinja2 template.
* Add CYCLE_DIR as an environment variable to the GET_EXTRN_ICS_TN and GET_EXTRN_LBCS_TN tasks.
* In the MAKE_ICS_TN, MAKE_LBCS_TN, RUN_FCST_TN, and RUN_POST_TN tasks, change the way CYCLE_DIR is set so that it does not include the ensemble member subdirectory.
* In the MAKE_ICS_TN, MAKE_LBCS_TN, RUN_FCST_TN, and RUN_POST_TN tasks, create the new environment variable SLASH_ENSMEM_SUBDIR that gets set to a null string if not running ensembles and to the string "/${name_of_ensemble_member}" when running ensembles.

* Bug fixes.  These bugs were introduced during the previous merge of the develop branch into this fork (feature/ensemble).

* To test the multiple-days and multiple-cycle-hours capabilities with ensemble forecasts in community mode, change WE2E test "community_ensemble_008" to include 2 days and 2 cycle hours per day (instead of 1 and 1, respectively).

* Add WCOSS changes for the feature/ensemble branch

* Minor changes to code comments.

Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@m71a3.ncep.noaa.gov>

* updated codes according to the comments
modified files:
       scripts/exregional_get_extrn_mdl_files.sh
       ush/NOMADS_get_extrn_mdl_files.sh
       ush/config_defaults.sh
       ush/generate_FV3SAR_wflow.sh
       ush/valid_param_vals.sh

* Change filename:
 ush/generate_FV3SAR_wflow.sh -> ush/generate_FV3LAM_wflow.sh

* adding one test to WE2W for downloading files
new file for the test:
        tests/baseline_configs/config.user_download_extrn_files.sh
modified file for the test:
        tests/baselines_list.txt
modified files for recent changes(SAR to LAM, JPgrid to ESGgrid)
        scripts/exregional_get_extrn_mdl_files.sh
        ush/config_defaults.sh
        ush/valid_param_vals.sh

Co-authored-by: Linlin.Pan <Linlin.Pan@noaa.gov>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@m71a3.ncep.noaa.gov>
EricJames-NOAA added a commit to EricJames-NOAA/UPP that referenced this issue Dec 14, 2022
…MC#249)

* Changes to exregional_run_post.sh required to run new EMC UPP.  Script changes involve new namelist syntax in itag file.

* Indenting model_inputs namelist lines per suggestion from Guoqing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants