-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add utility to compare Fortran namelists #1234
Merged
WalterKolczynski-NOAA
merged 5 commits into
NOAA-EMC:develop
from
aerorahul:feature/compare-fortran-namelists
Jan 11, 2023
Merged
Add utility to compare Fortran namelists #1234
WalterKolczynski-NOAA
merged 5 commits into
NOAA-EMC:develop
from
aerorahul:feature/compare-fortran-namelists
Jan 11, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalterKolczynski-NOAA
requested changes
Jan 11, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few things, plus docstrings need arguments/returns (I like the descriptions).
WalterKolczynski-NOAA
approved these changes
Jan 11, 2023
WalterKolczynski-NOAA
pushed a commit
that referenced
this pull request
Jan 11, 2023
Often times it is necessary to compare Fortran namelists between a UFS-weather-model regression test and a global-workflow experiment, or in other example applications. This PR adds a simple utility that loads two namelists and spits out the differences between them. The differences are calculated as a departure from the first namelist. This utility leverages `f90nml` (approved for use on WCOSS2) The usage is as follows: ``` ❯❯❯ python3 compare_f90nml.py -h usage: compare_f90nml.py [-h] [-r] left_namelist right_namelist Compare two Fortran namelists and display differences (left_namelist - right_namelist) positional arguments: left_namelist Left namelist to compare right_namelist Right namelist to compare options: -h, --help show this help message and exit -r, --reverse reverse diff (right_namelist - left_namelist) (default: False) ``` The comparison is done as follows: - Both namelists are loaded - We loop over the keys of `left_namelist`. We look for the same key in the `right_namelist`. If the key is found, the values are compared. If the key is not found, a note is made that the key is undefined in `right_namelist`. - Differences in the values are printed to screen. - The `-r | --reverse` reverses the `namelists`. This allows the user to use `right_namelist` as the reference. If differences are found, they are shown as follows (examples of `input.nml` from the `control_p8` and `cpld_control_p8` regression tests of the ufs-weather-model) ``` ❯❯❯ python3 compare_f90nml.py control_p8.nml cpld_control_p8.nml comparing: control_p8.nml | cpld_control_p8.nml ----------------------------------------------- atmos_model_nml: ccpp_suite : FV3_GFS_v17_p8 | FV3_GFS_v17_coupled_p8 fms_nml: domains_stack_size : 3000000 | 8000000 fv_core_nml: dnats : 0 | 2 gfs_physics_nml: min_seaice : 0.15 | 1e-06 use_cice_alb : False | True nstf_name : [2, 1, 0, 0, 0] | [2, 0, 0, 0, 0] cplchm : False | True cplflx : False | True cplice : False | True cplwav : False | True cplwav2atm : False | True ```
EdwardSafford-NOAA
pushed a commit
to EdwardSafford-NOAA/global-workflow
that referenced
this pull request
Jan 11, 2023
Adds support to running GDASapp portion of global-workflow in a container `CONTAINER.env` is quite minimalist at this point, it could also certainly be cleaned up, but it will need to be expanded later anyway to handle additional jobs. Fixes NOAA-EMC#1234
KateFriedman-NOAA
added a commit
to KateFriedman-NOAA/global-workflow
that referenced
this pull request
Jan 30, 2023
* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)
7 tasks
7 tasks
21 tasks
EricSinsky-NOAA
added a commit
that referenced
this pull request
Oct 11, 2024
<!-- PLEASE READ --> <!-- Any PRs not following this template will be closed --> <!-- Please use a short (<60 char), descriptive title above. It should complete the sentence "If merged, this PR will _____". Capitalize the first word and do not end with a period. PRs should meet these guidelines: - Each PR should address ONE topic and have an associated issue. - No hard-coded paths or personal directories. - No temporary or backup files should be committed (including logs). - Any code that you disabled by being commented out should be removed or reenabled. Please delete all these comments before submitting the PR. --> # Description <!-- This description will become the commit message for the PR--> <!-- Please use this format for your description: Describe your changes. Focus on the *what* and *why*. The *how* will be evident from the changes. In particular, be sure to note any interface changes, such as command line syntax, that will need to be communicated to users. At the end of your description, please be sure to add the issue this PR solves using the word "Resolves". If there are any issues that are related but not yet resolved (including in other repos), you may use "Refs". Resolves #1234 Refs #4321 Refs NOAA-EMC/repo#5678 --> This PR adds the repair task for correcting the F03 and F06 ave/acc/min/max variables. # Type of change <!-- Delete all except one --> - New feature (adds functionality) # Change characteristics - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.) - [ ] EMC verif-global - [ ] GDAS - [x] GFS-utils - [ ] GSI - [ ] GSI-monitor - [ ] GSI-utils - [ ] UFS-utils - [ ] UFS-weather-model - [ ] wxflow # How has this been tested? <!-- Please list any test you conducted, including the machine. Example: - Clone and build on WCOSS - Cycled test on Orion - Forecast-only on Hera --> This has been tested in WCOSS2. # Checklist - [ ] Any dependent changes have been merged and published - [ ] My code follows the style guidelines of this project - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] My changes generate no new warnings - [ ] New and existing tests pass with my changes - [ ] I have made corresponding changes to the documentation if necessary --------- Co-authored-by: Eric.Sinsky <eric.sinsky@noaa.gov>
EricSinsky-NOAA
added a commit
that referenced
this pull request
Oct 28, 2024
<!-- *** PLEASE READ *** Any PRs not following this template will be closed. Please delete all these comments before submitting the PR. Please use a short (<60 char), descriptive title for the PR title above. It should complete the sentence "If merged, this PR will _____". Capitalize the first word and do not end with a period. No content should appear above the "Description" header. If this PR is not merge-ready (e.g. it depends on other PRs not yet merged), please mark it as draft until it is ready. PRs should meet these guidelines: - Each PR should address ONE topic and have an associated issue. - No hard-coded paths or personal directories. - No temporary or backup files should be committed (including logs). - Any code that you disabled by being commented out should be removed or reenabled. --> # Description <!-- This description will become the commit message for the PR. --> <!-- Solely pointing to an issue is not an adequate description! Please use this format for your description: Describe your changes. Focus on the *what* and *why*. The *how* will be evident from the changes. In particular, be sure to note any interface changes, such as command line syntax, that will need to be communicated to users. At the end of your description, please be sure to add the issue this PR solves using the word "Resolves". If there are any issues that are related but not yet resolved (including in other repos), you may use "Refs". Resolves #1234 Refs #4321 Refs NOAA-EMC/repo#5678 --> This PR brings recent changes from the develop branch to the GEFS reforecast branch. This PR updates the GEFS reforecast branch to develop hash ac3cde5 (10/11/2024). This version of global-workflow uses the ufs-weather-model hash [6a4e09e](https://github.com/ufs-community/ufs-weather-model/tree/6a4e09e94773ffa39ce7ab6a54a885efada91f21) (9/9/2024). Furthermore, this PR ensures the following adjustments for the reforecast: - [x] Speed up rocoto by grouping post job - [x] Optimize PE configuration - [x] Remove duplicate OCNSPPT and EPBL settings - [x] Set restart_interval to fhmax - [x] Turn off SHUM in config.efcs - [x] Set FHMIN_WAV to 3 in config.base - [x] Turn off ATM history file output - [x] Change HMS=${cyc}0000 to HMS=030000 in Wavepostpnt script (#2788) - [x] Include YYYYMMDDHH (PDY) in job name - [x] Change CA seed based on case and cyc for control member and perturbed members - [x] Fix post ensemble info - [x] Add tob to ocean products (#2995 ) - [x] Move PEVPR from b group to a group for atmos products (#2995) - [x] Add option to download initial condition from HPSS - [x] Add ability to download and stage replay analysis from AWS, which is needed for the repair_replay task - [x] Add capability to run forecasts in 7-day intervals (#2928) - [x] Update defaults.yaml so that many of the reforecast-specific settings can be used by default <!-- For more on writing good commit messages, see https://cbea.ms/git-commit/ --> # Type of change - [ ] Bug fix (fixes something broken) - [ ] New feature (adds functionality) - [x] Maintenance (code refactor, clean-up, new CI test, etc.) # Change characteristics <!-- Choose YES or NO from each of the following and delete the other --> - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? NO - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 --> - [ ] GDAS <!-- NOAA-EMC/GDASApp#1234 --> - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 --> - [ ] GSI <!-- NOAA-EMC/GSI#1234 --> - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 --> - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 --> - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 --> - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 --> - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 --> # How has this been tested? <!-- Please list any test you conducted, including the machine. Example: - Clone and build on WCOSS - Cycled test on Orion - Forecast-only on Hera --> This branch is being tested on WCOSS2. When testing has succeeded, this PR will be marked as ready for review. # Checklist - [ ] Any dependent changes have been merged and published - [ ] My code follows the style guidelines of this project - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have documented my code, including function, input, and output descriptions - [ ] My changes generate no new warnings - [ ] New and existing tests pass with my changes - [ ] This change is covered by an existing CI test or a new one has been added - [ ] I have made corresponding changes to the system documentation if necessary --------- Co-authored-by: Wei Huang <wei.huang@noaa.gov> Co-authored-by: Kate Friedman <kate.friedman@noaa.gov> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Andrew.Tangborn <Andrew.Tangborn@noaa.gov> Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov> Co-authored-by: AndrewEichmann-NOAA <58948505+AndrewEichmann-NOAA@users.noreply.github.com> Co-authored-by: DavidBurrows-NCO <82525974+DavidBurrows-NCO@users.noreply.github.com> Co-authored-by: AnningCheng-NOAA <48297505+AnningCheng-NOAA@users.noreply.github.com> Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com> Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> Co-authored-by: AntonMFernando-NOAA <167725623+AntonMFernando-NOAA@users.noreply.github.com> Co-authored-by: BoCui-NOAA <53531984+BoCui-NOAA@users.noreply.github.com> Co-authored-by: DavidNew-NOAA <134300700+DavidNew-NOAA@users.noreply.github.com> Co-authored-by: Jeffrey Whitaker <jeffrey.s.whitaker@noaa.gov> Co-authored-by: mingshichen-noaa <48537176+mingshichen-noaa@users.noreply.github.com> Co-authored-by: Jiarui Dong <Jiarui.Dong@noaa.gov> Co-authored-by: David Huber <david.huber@noaa.gov> Co-authored-by: Guillaume Vernieres <guillaume.vernieres@gmail.com> Co-authored-by: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com> Co-authored-by: Innocent Souopgui <162634017+InnocentSouopgui-NOAA@users.noreply.github.com> Co-authored-by: Neil Barton <103681022+NeilBarton-NOAA@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Often times it is necessary to compare Fortran namelists between a UFS-weather-model regression test and a global-workflow experiment, or in other example applications.
This PR adds a simple utility that loads two namelists and spits out the differences between them. The differences are calculated as a departure from the first namelist.
This utility leverages
f90nml
(approved for use on WCOSS2)The usage is as follows:
The comparison is done as follows:
left_namelist
. We look for the same key in theright_namelist
. If the key is found, the values are compared. If the key is not found, a note is made that the key is undefined inright_namelist
.The
-r | --reverse
reverses thenamelists
. This allows the user to useright_namelist
as the reference.If differences are found, they are shown as follows (examples of
input.nml
from thecontrol_p8
andcpld_control_p8
regression tests of the ufs-weather-model)Type of change
How Has This Been Tested?
As shown above in comparing 2 namelists from ufs-weather-model regression tests.
Checklist