Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rt update #261

Merged
merged 25 commits into from
Nov 19, 2020
Merged

rt update #261

merged 25 commits into from
Nov 19, 2020

Conversation

MinsukJi-NOAA
Copy link
Contributor

@MinsukJi-NOAA MinsukJi-NOAA commented Oct 31, 2020 •
edited
Loading

Description

Update rt test to:

Testing

RT will be performed on tier 1 platforms.

Dependencies

N/A

@MinsukJi-NOAA MinsukJi-NOAA added the Baseline Updates Current baselines will be updated. label Oct 31, 2020
@MinsukJi-NOAA MinsukJi-NOAA marked this pull request as ready for review November 1, 2020 19:05
@DeniseWorthen
Copy link
Collaborator

This looks like it should work for DATM also, as long as there was an rt.conf file for that use case, correct?

@MinsukJi-NOAA
Copy link
Contributor Author

This looks like it should work for DATM also, as long as there was an rt.conf file for that use case, correct?

Yes, you are right.

@DeniseWorthen
Copy link
Collaborator

I got an error when trying the 35d no wave test using ecflow using ./rt.sh -fek -l rt_35d.conf >output 2>&1 &

My test build directory is here: /scratch2/NCEPDEV/climate/Denise.Worthen/WORK/ufs_rt/tests

The only change I made was to edit the test to run only 2 days instead of 35d.

@MinsukJi-NOAA
Copy link
Contributor Author

@DeniseWorthen, are you using ecFlow and using your own executable? Not sure if that works...

@DeniseWorthen
Copy link
Collaborator

I did not try to build the executable first. This was just checking out your branch, editing the tests/cpld_bmark_35d for 2d instead of 35d and then launching the rt_35d.conf, including the build.

I always ran with ecflow when doing the 35d tests in ufs-s2s-model.

@MinsukJi-NOAA
Copy link
Contributor Author

I did not try to build the executable first. This was just checking out your branch, editing the tests/cpld_bmark_35d for 2d instead of 35d and then launching the rt_35d.conf, including the build.

I always ran with ecflow when doing the 35d tests in ufs-s2s-model.

Just realized that the 35d bmark tests are run on Orion only. 35 days with and without wave were tested on Orion for the full 35d period. Your error message on Hera seems to be related to the ecflow though.

@DeniseWorthen
Copy link
Collaborator

The 35d tests were restricted to orion when we had lost access to hera for the prototype runs. But that restriction is now lifted so I think we should be able to run them on either hera or orion.

But you are right, the current rt_35d.conf is restricted to compiling on orion. I added hera.intel and it appears to have resolved the first failure. I don't think the baselines have all the inputs they need on hera though.

@DeniseWorthen
Copy link
Collaborator

Actually the BM_IC does exist on hera. I've verified that the 35d test (edited for 2d) run there as long as hera.intel is added to the compile line.

DeniseWorthen
DeniseWorthen reviewed Nov 2, 2020
View reviewed changes
tests/rt_35d.conf Outdated Show resolved Hide resolved
aerorahul
aerorahul requested changes Nov 2, 2020
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments about hard-wired paths

tests/rt.sh Outdated Show resolved Hide resolved
tests/rt.sh Outdated Show resolved Hide resolved
tests/utest Outdated Show resolved Hide resolved
tests/utest Outdated Show resolved Hide resolved
tests/utest Outdated Show resolved Hide resolved
tests/rt.sh Outdated Show resolved Hide resolved
@junwang-noaa
Copy link
Collaborator

Minsuk, I see the nccmp is only applied to the coupled restart file, is it possible that we can apply it to all the nc files. Currently in the RT, I commented out the two parallel netcdf tests (regional one is in my PR#206) whose output files need to be compared with nccmp, and cmp comparison will show a failure. So to test then, please uncomment the two tests in rt.conf:

#RUN | fv3_ccpp_regional_quilt_netcdf_parallel | standard | | fv3 |
and
#RUN | fv3_ccpp_wrtGauss_netcdf_parallel | standard | | fv3 |

@MinsukJi-NOAA
Copy link
Contributor Author

Minsuk, I see the nccmp is only applied to the coupled restart file, is it possible that we can apply it to all the nc files. Currently in the RT, I commented out the two parallel netcdf tests (regional one is in my PR#206) whose output files need to be compared with nccmp, and cmp comparison will show a failure. So to test then, please uncomment the two tests in rt.conf:

#RUN | fv3_ccpp_regional_quilt_netcdf_parallel | standard | | fv3 |
and
#RUN | fv3_ccpp_wrtGauss_netcdf_parallel | standard | | fv3 |

fv3_ccpp_wrtGauss_netcdf_parallel_prod is still not able to pass.

baseline` dir = /work/noaa/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201022/INTEL/fv3_wrtGauss_netcdf_ccpp
working dir  = /work/noaa/stmp/jminsuk/stmp/jminsuk/FV3_RT/rt_1077/fv3_ccpp_wrtGauss_netcdf_parallel_prod
Checking test 002 fv3_ccpp_wrtGauss_netcdf_parallel results ....
 Comparing atmos_4xdaily.tile1.nc .........OK
 Comparing atmos_4xdaily.tile2.nc .........OK
 Comparing atmos_4xdaily.tile3.nc .........OK
 Comparing atmos_4xdaily.tile4.nc .........OK
 Comparing atmos_4xdaily.tile5.nc .........OK
 Comparing atmos_4xdaily.tile6.nc .........OK
 Comparing phyf000.nc ............NOT OK...TRY NCCMP.......NOT OK
 Comparing phyf024.nc ............NOT OK...TRY NCCMP.......NOT OK
 Comparing dynf000.nc ............NOT OK...TRY NCCMP.......NOT OK
 Comparing dynf024.nc ............NOT OK...TRY NCCMP.......NOT OK

baseline dir = /work/noaa/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201022/INTEL/fv3_regional_quilt_netcdf_parallel_ccpp
working dir  = /work/noaa/stmp/jminsuk/stmp/jminsuk/FV3_RT/rt_1077/fv3_ccpp_regional_quilt_netcdf_parallel_prod
Checking test 007 fv3_ccpp_regional_quilt_netcdf_parallel results ....
 Comparing atmos_4xdaily.nc .........OK
 Comparing dynf000.nc .........OK
 Comparing dynf024.nc ............NOT OK...TRY NCCMP......OK
 Comparing phyf000.nc .........OK
 Comparing phyf024.nc ............NOT OK...TRY NCCMP......OK
Test 007 fv3_ccpp_regional_quilt_netcdf_parallel PASS

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Nov 3, 2020 via email

@MinsukJi-NOAA
Use which to find nccmp path. Apply nccmp to all files that fail cmp.…
d0c964e 
… Fix a bug in fv3_ccpp_wrtGauss_netcdf_parallel
@MinsukJi-NOAA
Copy link
Contributor Author

Minsuk, Thanks for testing it. I realized one bug in fv3_ccpp_wrtGauss_netcdf_parallel_prod. Can you change the following line in tests/tests/fv3_ccpp_wrtGauss_netcdf_parallel https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/tests/fv3_ccpp_wrtGauss_netcdf_parallel export CNTL_DIR=fv3_wrtGauss_netcdf to export CNTL_DIR=fv3_wrtGauss_netcdf_parallel
…

Jun, that change made the test pass. I included the change in my latest commit.

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Nov 3, 2020 via email

aerorahul
aerorahul approved these changes Nov 3, 2020
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing comments.

@MinsukJi-NOAA
Copy link
Contributor Author

I did some comparison between nccmp and compare_ncfile.py. nccmp is okay with mediator restart files, but takes too long for netcdf_parallel files.
netcdf

@MinsukJi-NOAA MinsukJi-NOAA added the Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. label Nov 12, 2020
@aerorahul aerorahul self-requested a review November 12, 2020 19:26
@DeniseWorthen
Copy link
Collaborator

Are we ready to commit this? Shan is asking about running the 35d tests for frac/non-frac over the weekend and she would need this commit to be able to do that.

DusanJovic-NOAA
DusanJovic-NOAA reviewed Nov 13, 2020
View reviewed changes
tests/rt.sh Outdated Show resolved Hide resolved
@MinsukJi-NOAA MinsukJi-NOAA removed the Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. label Nov 13, 2020
@aerorahul aerorahul self-requested a review November 18, 2020 17:26
@MinsukJi-NOAA
Copy link
Contributor Author

This PR is good to go. CI issue is not addressed in this PR because of the hpc-stack issue

@MinsukJi-NOAA MinsukJi-NOAA added Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. Waiting for Reviews The PR is waiting for reviews from associated component PR's. labels Nov 19, 2020
@junwang-noaa junwang-noaa merged commit 248b609 into ufs-community:develop Nov 19, 2020
@MinsukJi-NOAA MinsukJi-NOAA mentioned this pull request Nov 19, 2020
Closed
@MinsukJi-NOAA MinsukJi-NOAA deleted the feature/rt-update branch November 23, 2020 14:12
AnningCheng-NOAA added a commit to AnningCheng-NOAA/ufs-weather-model that referenced this pull request Mar 8, 2021
@AnningCheng-NOAA
Merge remote-tracking branch 'upstream/develop' into merra2
e6bc1e4 
* upstream/develop:
  update MOM6 to GFDL 20210224 main branch commit (ufs-community#439)
  Add GNU and Cheyenne Support to Automated RT (ufs-community#444)
  Move Noah MP init to CCPP and update Noah MP regression tests, ice flux init bug fix in CCPP (ufs-community#425)
  Feature/rt automation (ufs-community#403)
  Update ccpp-physics. Make RRTMGP thread safe (ufs-community#418)
  Update regression tests from GFSv15+Thompson to GFSv16+Thompson, include "Add one regional regression test in DEBUG mode. (ufs-community#419)" (ufs-community#421)
  UGWP v0 v1 combined (ufs-community#396)
  add optional mesh in MOM6; add dz_min and min_seaice as configurable variables for coupled model (ufs-community#399)
  updates FMS to 2020.04.01 (ufs-community#392)
  Move LSM vegetation lookup tables into CCPP, clean up RUC snow cover on ice initialization (remove IPD step 2)  (ufs-community#407)
  Update CMEPS for HAFS integration; add datm and coupled-model tests on Gaea (ufs-community#401)
  Remove legacy gnumake build from fv3atm and NEMS, remove legacy Python 2.7 support, rename v16beta to v16 and RT updates (ufs-community#384)
  MOM6 bugfixes, GFDL update, update CDMBGWD settings; fix for restart reproducibility (without waves) when USE_LA_LI2016=True, sign error on fprec passed to ocean, GFDL update, resolution dependent cdmbgwd settings (ufs-community#379)
  dycore options to add zero-gradient BC to reconstruct interface u/v and change dz_min as input (ufs-community#369)
  Update develop from NOAA-GSL: RUC ice, MYNN sfclay, stochastic land perturbations (ufs-community#386)
  update cpl gfsv16 tests, rrtmgp fix and bug fixes in cmeps (ufs-community#378)
  point fv3 to EMC develop branch (ufs-community#377)
  Remove IPD steps 3 and 5 (ufs-community#357)
  Update CMEPS  (ufs-community#345)
  Implementation of CCPP timestep_init and timestep_final phases (ufs-community#337)
  Remove unnecessary SIMD instruction sets for Jet, first round of cleanup in rt.conf, initialize cld_amt to zero for regional runs (dycore) (ufs-community#353)
  add frac grid input, update and add additional cpld tests (ufs-community#354)
  Add checkpoint restarts for ufs-cpld (ufs-community#342)
  Update the format of rt.conf (ufs-community#349)
  Remove IPD (step 1) (ufs-community#331)
  Feature/ww3update (ufs-community#334)
  Replace old regional SDF with FV3_GFS_v15_thompson_mynn (ufs-community#333)
  Update modules with hpc-stack v1.1.0 (ufs-community#319)
  Regression test log for PR ufs-community#323 for jet.intel (ufs-community#336)
  RRTMGP and Thompson MP coupling (ufs-community#323)
  Add 2 new tests for DATM-MOM6-CICE6 application (ufs-community#332)
  Add optional bulk flux calculation in ufs-datm (ufs-community#266)
  Final-final GFS v16 updates / restart reproducibility bugfixes (ufs-community#325)
  Updates to build for JEDI linking/control, add wcoss2 (ufs-community#295)
  Update CICE, Move regression test input outside baseline directory (ufs-community#270)
  Feature/update mom6 and retain b4b results for 025x025 resolution (ufs-community#290)
  Update for Jet, bug fixes in running with frac_grid=T and GFDL MP, and in restarting with frac_grid=T  (ufs-community#304)
  Updates to stochastic_physics_wrapper (ufs-community#280)
  Update develop from gsd/develop 2020/11/20: Unified gravity wave drag, updates to other GSL physics (ufs-community#297)
  Fix to allow quilting with non-factors for layout (ufs-community#250)
  rt update (ufs-community#261)
pjpegion pushed a commit to NOAA-PSL/ufs-weather-model.p7b that referenced this pull request Jul 20, 2021
@climbfuji
Wrapper PR for ufs-community#261, ufs-community#271, ufs-community#268:…
0dde422 
… zorl interstitial, ocn -> wat, merra2 threading (ufs-community#279)

* changed .gitmodules to point to merra2 ccpp/physics
* remove GFDL_atmos_cubed_sphere and ccpp-framework from .git module
* remove IPD gfsphysics
* Update .gitmodules and submodule pointer for ccpp-physics for code review and testing
* Remove interstitial zorl composites
* Update .gitmodules and submodule pointer fpor ccpp-physics for code review and testing
* Remove or replace references to IPD in comments in atmos_model.F90
* Initialize Sfcprop%zorlx to clear_val instead of huge
* Update submodule pointer for ccpp-physics
* Rename Fortran variables and CCPP standard names / long names of surface composites from ocean to water
* Rename Sfcprop%zorlw to Sfcprop%zorlwav
* Rename Sfcprop%zorlo to Sfcprop%zorlw
* update submodule pointer for ccpp-physics
* Revert change to .gitmodules and update submodule pointer for ccpp-physics
Co-authored-by: anning.cheng <anning.cheng@noaa.gov>
epic-cicd-jenkins pushed a commit that referenced this pull request Apr 17, 2023
@chan-hoo
update hash of regional workflow (#261)
f63b836
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DeniseWorthen DeniseWorthen DeniseWorthen approved these changes

@DusanJovic-NOAA DusanJovic-NOAA DusanJovic-NOAA approved these changes

@junwang-noaa junwang-noaa junwang-noaa approved these changes

@climbfuji climbfuji climbfuji approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees
No one assigned
Labels
Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. Waiting for Reviews The PR is waiting for reviews from associated component PR's.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@MinsukJi-NOAA @DeniseWorthen @junwang-noaa @aerorahul @DusanJovic-NOAA @climbfuji