Rt hires #1471

RatkoVasic-NOAA · 2022-10-25T15:22:32Z

PR Checklist

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
Issue Replacing regional regression tests with more realistic ones #1461
Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.
New tests make new results.
Should create new develop-2022XXXX directory
New input data is required and created on Hera machine. Should be resynched to other platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsibility to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

Replaced old regional regression tests with new, using latest configuration from LAMDX parallel. It is in 3km resolution over small domain.
@DusanJovic-NOAA removed variable "calendar" from namelists (model_configure*.IN) and replaced calendar variable values in input directory in files coupler.res from 2 --> 3
(Calendar: no_calendar=0, thirty_day_months=1, julian=2, gregorian=3, noleap=4)

Issue(s) addressed

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

Dependencies

No dependences

DusanJovic-NOAA · 2022-10-26T13:10:37Z

I think we can remove the following directories from the new input-data:

FV3_input_data_gsd/FV3_input_data_C96_with_aerosols/
FV3_input_data_regional_esg/
FV3_input_data_sar/
GOCART/p7/
GOCART/ExtData/
WW3_input_data_20211113/
WW3_input_data_20220418/

most of the files from AQM/NEXUS - confirmed by @BrianCurtis-NOAA only NEXUS_Expt.nc is needed

RatkoVasic-NOAA · 2022-10-26T13:16:46Z

I an remove those directories from my input-data-20221101 directory on Hera and run full tests.
Who can confirm that directories GOCART and WW3 are not in use?

RatkoVasic-NOAA · 2022-10-26T15:14:19Z

I removed those directories and ran tests on Hera. Tests PASSED.

DusanJovic-NOAA

tests/rt.sh file permissions has been changed from 755 to 644. Please revert.

RatkoVasic-NOAA · 2022-10-26T17:57:05Z

tests/rt.sh file permissions has been changed from 755 to 644. Please revert.

I guess it happens when you solve conflict using github's web based tools.

jkbk2004 · 2022-11-04T13:14:28Z

@RatkoVasic-NOAA I may start rsync if input-data-20221101 is ready on hera.

RatkoVasic-NOAA · 2022-11-04T13:45:32Z

@jkbk2004 I see two new directories in input-data-20220414 :
DATM_GSWP3_input_data
NOAHMP_IC
Are they added recently?
Should I add those to input-data-20221101?

RatkoVasic-NOAA · 2022-11-04T13:50:53Z

@jkbk2004
OK, I see they are used in datm_cdeps_lnd_gswp3 and datm_cdeps_lnd_gswp3_rst.
I'll copy those directories on Hera and WCOSS2, and you can rsync to othermachines.

uturuncoglu · 2022-11-04T15:03:53Z

@RatkoVasic-NOAA Yes, those are added as a part of external land component PR.

DusanJovic-NOAA · 2022-11-09T17:30:31Z

With the above change in fv3_slurm.IN_jet the regional_2threads run finished in ~260sec.

DusanJovic-NOAA · 2022-11-09T17:31:24Z

I pushed the change to Ratko's branch. Please rerun the test on Jet.

jkbk2004 · 2022-11-09T17:44:31Z

I pushed the change to Ratko's branch. Please rerun the test on Jet.

@DusanJovic-NOAA Great! do you think --npernode will same thing for cheyenne's mpiexec_mpt? let me try.

BrianCurtis-NOAA · 2022-11-09T17:46:18Z

I pushed the change to Ratko's branch. Please rerun the test on Jet.

@DusanJovic-NOAA is this something we should be doing on all machines we run that test on?

DusanJovic-NOAA · 2022-11-09T21:13:07Z

I pushed the change to Ratko's branch. Please rerun the test on Jet.

@DusanJovic-NOAA is this something we should be doing on all machines we run that test on?

We can test this change on other machines that are using slurm scheduler.

DusanJovic-NOAA · 2022-11-09T22:07:20Z

I pushed the change to Ratko's branch. Please rerun the test on Jet.

@DusanJovic-NOAA is this something we should be doing on all machines we run that test on?

We can test this change on other machines that are using slurm scheduler.

I added --cpus-per-task=@[THRD] to hera job card template and ran 4 tests, control, control_2threads, regional and regional_2threads, and all tests passed.

jkbk2004 · 2022-11-10T00:47:03Z

I am wondering one issue about the setup of TASKS/INPES/JNPES of regional_control/regional_2thread. On cheyenne, regional_control job_card sets #PBS -l select=4:ncpus=36:mpiprocs=36 and mpiexec_mpt -p %g: -np 144 ./fv3.exe and
regional_2thread job_card sets #PBS -l select=7:ncpus=18:mpiprocs=18 and mpiexec_mpt -p %g: -np 126 ./fv3.exe. This doesn't makes a sense. Another problem is both cases set with export TASKS=120. The numbers don't match. @RatkoVasic-NOAA @DusanJovic-NOAA @BrianCurtis-NOAA It could be a bug on setting the cases on cheyenne.

RatkoVasic-NOAA · 2022-11-10T01:11:20Z

I don't know who set up cheyenne parameters originally. Most of us here don't have access to that machine.
Yes, those numbers look odd.
Can we just skip that test on cheyenne in rt.conf?

jkbk2004 · 2022-11-10T01:24:28Z

I agree to turn off regional_2thread on cheyenne and move on for merging. I am going to create an issue about this. If all agrees, then probably we can start merging tomorrow morning. @DusanJovic-NOAA @BrianCurtis-NOAA what do you think?

DusanJovic-NOAA · 2022-11-10T01:26:16Z

I am wondering one issue about the setup of TASKS/INPES/JNPES of regional_control/regional_2thread. On cheyenne, regional_control job_card sets #PBS -l select=4:ncpus=36:mpiprocs=36 and mpiexec_mpt -p %g: -np 144 ./fv3.exe and regional_2thread job_card sets #PBS -l select=7:ncpus=18:mpiprocs=18 and mpiexec_mpt -p %g: -np 126 ./fv3.exe. This doesn't makes a sense. Another problem is both cases set with export TASKS=120. The numbers don't match. @RatkoVasic-NOAA @DusanJovic-NOAA @BrianCurtis-NOAA It could be a bug on setting the cases on cheyenne.

The number of processes (-np) is calculated by multiplying number of nodes with number_of_tasks_per_node. So for regional_control that is 4*36 = 144, for regional_2threads it is 7*18 = 126.

wcoss2, is also using PBS. Try to set some of the pbs options specified there, specifically 'place=vscatter' and mpiexec options '-ppn @[TPN] -depth @[THRD]'. I do not know if mpiexec_mpt ob cheyenne is the same as mpiexec on wcoss.

I also do not have access to cheyenne.

BrianCurtis-NOAA · 2022-11-10T13:24:08Z

Let me give it a go on Cheyenne before we write off the RT on Cheyenne.

BrianCurtis-NOAA · 2022-11-10T16:26:46Z

diff --git a/tests/fv3_conf/fv3_qsub.IN_cheyenne b/tests/fv3_conf/fv3_qsub.IN_cheyenne
index 2cc648b8..b87293ce 100644
--- a/tests/fv3_conf/fv3_qsub.IN_cheyenne
+++ b/tests/fv3_conf/fv3_qsub.IN_cheyenne
@@ -4,7 +4,7 @@
 #PBS -N @[JBNME]
 #PBS -A @[ACCNR]
 #PBS -q @[QUEUE]
-#PBS -l select=@[NODES]:ncpus=@[TPN]:mpiprocs=@[TPN]
+#PBS -l select=@[NODES]:ncpus=@[TPN]:mpiprocs=@[TPN]:ompthreads=@[THRD]
 #PBS -l walltime=00:@[WLCLK]:00
 
 set -eux
@@ -34,7 +34,7 @@ export ESMF_RUNTIME_PROFILE_OUTPUT="SUMMARY"
 # Avoid job errors because of filesystem synchronization delays
 sync && sleep 1
 
-mpiexec_mpt -p %g: -np @[TASKS] ./fv3.exe
+mpiexec_mpt -p %g: -np @[TASKS] omplace ./fv3.exe
 
 echo "Model ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

BrianCurtis-NOAA · 2022-11-10T16:28:00Z

@jkbk2004 Please try those changes with a few tests and re-run the regional_2threads as well for the logs.

jkbk2004 · 2022-11-10T16:37:21Z

@jkbk2004 Please try those changes with a few tests and re-run the regional_2threads as well for the logs.

sure!

jkbk2004 · 2022-11-10T19:10:24Z

With @BrianCurtis-NOAA contribution of omplace, regional_2threads passes ok now. @DusanJovic-NOAA @RatkoVasic-NOAA @ChunxiZhang-NOAA this pr is ready.

RatkoVasic-NOAA added 3 commits October 21, 2022 20:16

Replace low resolution regression tests with high resolution.

8b9e140

Rename ICs and result directory

750e9a7

Remove "calendar" variable from namelists.

dd0d109

RatkoVasic-NOAA requested a review from DusanJovic-NOAA October 25, 2022 15:22

DeniseWorthen added Baseline Updates Current baselines will be updated. New Input Data Req'd This PR requires new data to be sync across platforms labels Oct 25, 2022

Tests passed on Hera.

da11850

RatkoVasic-NOAA and others added 3 commits October 26, 2022 16:15

Rename regional tests from fv3* to FV3*

8ad0466

Hera - log file

b475c8b

Merge branch 'develop' into rt-hires

39d8717

DusanJovic-NOAA reviewed Oct 26, 2022

View reviewed changes

Chmod rt.sh

3e95d8b

RatkoVasic-NOAA added 8 commits October 31, 2022 16:47

Merge branch 'develop' into rt-hires

4adf449

Remove unused scripts: gfdlmp_run.IN & gsd_run.IN

46c9b47

Merge

4fe9704

Merge branch 'develop' into rt-hires

93e728d

Rename new wofs RTs

aa6ce36

Revert WW3

474b8e6

Revert FV3

35e2c89

Merge branch 'develop' into rt-hires

9be505d

DeniseWorthen mentioned this pull request Nov 7, 2022

RRFS Q4FY2022 #1331

Closed

add new BL_DATE

e920d43

Acorn Intel RT Log

d192972

add jet.intel RT log: passed

5ee576f

jkbk2004 mentioned this pull request Nov 10, 2022

regional_2threads hanging issue on cheyenne #1485

Closed

update INPUTDATA_ROOT: input-data-20221101

9872bac

jkbk2004 added 2 commits November 10, 2022 12:01

add cheyenne.intel RT log: passed

da66a9a

add omplace to cheyenne qsub script

62d8eee

jkbk2004 requested review from BrianCurtis-NOAA, ChunxiZhang-NOAA and jkbk2004 November 10, 2022 19:06

jkbk2004 approved these changes Nov 10, 2022

View reviewed changes

DusanJovic-NOAA approved these changes Nov 10, 2022

View reviewed changes

BrianCurtis-NOAA approved these changes Nov 10, 2022

View reviewed changes

jkbk2004 merged commit ad8da86 into ufs-community:develop Nov 10, 2022

RatkoVasic-NOAA deleted the rt-hires branch November 10, 2022 19:50

grantfirl mentioned this pull request Nov 16, 2022

UFS-dev PR#16 NCAR/ufs-weather-model#76

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rt hires #1471

Rt hires #1471

RatkoVasic-NOAA commented Oct 25, 2022 •

edited by jkbk2004

Loading

DusanJovic-NOAA commented Oct 26, 2022 •

edited

Loading

RatkoVasic-NOAA commented Oct 26, 2022

RatkoVasic-NOAA commented Oct 26, 2022

DusanJovic-NOAA left a comment

RatkoVasic-NOAA commented Oct 26, 2022

jkbk2004 commented Nov 4, 2022

RatkoVasic-NOAA commented Nov 4, 2022

RatkoVasic-NOAA commented Nov 4, 2022

uturuncoglu commented Nov 4, 2022

DusanJovic-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

jkbk2004 commented Nov 9, 2022

BrianCurtis-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

jkbk2004 commented Nov 10, 2022

RatkoVasic-NOAA commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

DusanJovic-NOAA commented Nov 10, 2022 •

edited

Loading

BrianCurtis-NOAA commented Nov 10, 2022

BrianCurtis-NOAA commented Nov 10, 2022

BrianCurtis-NOAA commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

Rt hires #1471

Rt hires #1471

Conversation

RatkoVasic-NOAA commented Oct 25, 2022 • edited by jkbk2004 Loading

PR Checklist

Instructions: All subsequent sections of text should be filled in as appropriate.

Description

Issue(s) addressed

Testing

Dependencies

DusanJovic-NOAA commented Oct 26, 2022 • edited Loading

RatkoVasic-NOAA commented Oct 26, 2022

RatkoVasic-NOAA commented Oct 26, 2022

DusanJovic-NOAA left a comment

Choose a reason for hiding this comment

RatkoVasic-NOAA commented Oct 26, 2022

jkbk2004 commented Nov 4, 2022

RatkoVasic-NOAA commented Nov 4, 2022

RatkoVasic-NOAA commented Nov 4, 2022

uturuncoglu commented Nov 4, 2022

DusanJovic-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

jkbk2004 commented Nov 9, 2022

BrianCurtis-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

DusanJovic-NOAA commented Nov 9, 2022

jkbk2004 commented Nov 10, 2022

RatkoVasic-NOAA commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

DusanJovic-NOAA commented Nov 10, 2022 • edited Loading

BrianCurtis-NOAA commented Nov 10, 2022

BrianCurtis-NOAA commented Nov 10, 2022

BrianCurtis-NOAA commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

jkbk2004 commented Nov 10, 2022

RatkoVasic-NOAA commented Oct 25, 2022 •

edited by jkbk2004

Loading

DusanJovic-NOAA commented Oct 26, 2022 •

edited

Loading

DusanJovic-NOAA commented Nov 10, 2022 •

edited

Loading