Use inline post with cubed sphere history output #1831

DusanJovic-NOAA · 2023-07-11T16:41:08Z

Description

Add new model_configure parameter (history_file_on_native_grid) to allow history outputs on native (cubed sphere) grid to be created when inline post is enabled.

Fixes: #1803

Input data additions/changes

No changes are expected to input data.
Changes are expected to input data:
- New input data.
- Updated input data.

Anticipated changes to regression tests:

No changes are expected to any regression test.
Changes are expected to the following tests: control_CubedSphereGrid_parallel

Subcomponents involved:

Library Updates/Changes

Not Needed
Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
Add issue link from JCSDA/spack-stack following this item

Combined with PR's (If Applicable):

Commit Queue Checklist:

Link PR's from all sub-components involved in section below
Confirm reviews completed in ALL sub-component PR's
Add all appropriate labels to this PR.
Run full RT suite on either Hera/Cheyenne AND attach log to a PR comment.
Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Depends on Use inline post with cubed sphere history output NOAA-EMC/fv3atm#680

Testing Day Checklist:

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

RDHPCS
- Hera
- Orion
- Jet
- Gaea
- Cheyenne
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

…ry_output

DusanJovic-NOAA · 2023-07-11T16:44:17Z

RegressionTests_hera.log

junwang-noaa · 2023-07-11T17:24:48Z

@DusanJovic-NOAA do you want to add test to use the option history_file_on_native_grid?

DusanJovic-NOAA · 2023-07-11T18:05:27Z

@DusanJovic-NOAA do you want to add test to use the option history_file_on_native_grid?

I don't know, should we add new test? We already have so many tests.

junwang-noaa · 2023-07-11T19:57:22Z

I'd suggest of adding a low resolution one, just to make sure the feature is working to support DA development.

DusanJovic-NOAA · 2023-07-11T20:40:56Z

How about changing the control_CubedSphereGrid_parallel test, by enabling inline post, which is currently turned off, and adding a new option to the model_configure to create native grid output files?

The control_CubedSphereGrid_parallel test has been updated to run post and create native grid history files

FernandoAndrade-NOAA · 2023-07-25T17:26:02Z

@DusanJovic-NOAA, would you be ok with introducing the changes to the remaining tests in #1832 into this PR?
The two remaining would be

hrrr_control_qr_dyn32_phy32 (intel)
hrrr_control_qr_dyn32_phy32 (gnu)

…ry_output

DusanJovic-NOAA · 2023-07-25T19:26:18Z

I changed two hrrr_control_qr_dyn32_phy32 tests to compare against the same baseline as hrrr_control_dyn32_phy32 and ran regression test but it failed:

$ cat logs/log_hera/rt_002_hrrr_control_qr_dyn32_phy32_intel.log 

baseline dir = /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20230720/hrrr_control_dyn32_phy32_intel
working dir  = /scratch1/NCEPDEV/stmp2/Dusan.Jovic/FV3_RT/rt_270108/hrrr_control_qr_dyn32_phy32_intel
Checking test 002 hrrr_control_qr_dyn32_phy32_intel results ....
 Comparing sfcf000.nc .........OK
 Comparing sfcf009.nc .........OK
 Comparing sfcf012.nc .........OK
 Comparing atmf000.nc .........OK
 Comparing atmf009.nc .........OK
 Comparing atmf012.nc .........OK
 Comparing GFSFLX.GrbF00 .........OK
 Comparing GFSFLX.GrbF09 .........OK
 Comparing GFSFLX.GrbF12 .........OK
 Comparing GFSPRS.GrbF00 .........OK
 Comparing GFSPRS.GrbF09 .........OK
 Comparing GFSPRS.GrbF12 .........OK
 Comparing RESTART/20210322.120000.coupler.res .........OK
 Comparing RESTART/20210322.120000.fv_core.res.nc .........OK
 Comparing RESTART/20210322.120000.fv_core.res.tile1.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_core.res.tile2.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_core.res.tile3.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_core.res.tile4.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_core.res.tile5.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_core.res.tile6.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile1.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile2.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile3.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile4.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile5.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_srf_wnd.res.tile6.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile1.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile2.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile3.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile4.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile5.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.fv_tracer.res.tile6.nc ............ALT CHECK......OK
 Comparing RESTART/20210322.120000.phy_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.phy_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.phy_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.phy_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.phy_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.phy_data.tile6.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/20210322.120000.sfc_data.tile6.nc ............ALT CHECK......NOT OK

  0: The total amount of wall time                        = 193.250397
  0: The maximum resident set size (KB)                   = 968648

Test 002 hrrr_control_qr_dyn32_phy32_intel FAIL

All history outputs, grib files and dynamics restart files are identical, but physics are not. This is not expected.

This will require more debugging, unfortunately we can not combine these changes with this PR at this moment.

FernandoAndrade-NOAA · 2023-07-25T19:59:30Z

Understood, thanks for verifying! We'll begin testing for this PR after 1811 gets merged in.

FernandoAndrade-NOAA · 2023-07-26T20:45:33Z

Understood, thanks for verifying! We'll begin testing for this PR after 1811 gets merged in.

Apologies for any confusion, there's been some shift in the commit queue testing order: 1794 current, 1825, and then this PR. I'll post an update once we're ready to work on this PR.

…ry_output

FernandoAndrade-NOAA · 2023-07-28T15:25:02Z

Ok, #1825 has been merged in, please go ahead and sync up your branches and any resolve conflicts. We'll begin testing shortly after.
@DusanJovic-NOAA

…ry_output

FernandoAndrade-NOAA · 2023-07-28T15:58:48Z

@BrianCurtis-NOAA Getting started with jenkins ci for this PR

FernandoAndrade-NOAA · 2023-07-28T18:55:52Z

Jenkins ci log attached, ORTs passed.
ufs-weather-model » ort-docker-pipeline » PR-1831 #1 Console [Jenkins].txt

zach1221 · 2023-07-31T10:51:49Z

Good morning, @DusanJovic-NOAA . Can you can that I'm added to your branch here as a contributor? Trying to push logs to this PR, and am being told I'm not authorized.

zach1221 · 2023-07-31T12:01:54Z

Good morning, @DusanJovic-NOAA . Can you can that I'm added to your branch here as a contributor? Trying to push logs to this PR, and am being told I'm not authorized.

Disregard, problem on my end.

zach1221 · 2023-07-31T18:54:37Z

Testing is complete. I will follow up on fv3atm sub-pr #680 .

zach1221 · 2023-07-31T19:19:58Z

@DusanJovic-NOAA fv3atm PR #680 has been merged. Please update the sub-module pointer and revert the changes in gitmodules
fv3atm hash: f595b972ab0a099a8ef13ddad313ec2c0d8672d4

DusanJovic-NOAA · 2023-07-31T19:25:48Z

@DusanJovic-NOAA fv3atm PR #680 has been merged. Please update the sub-module pointer and revert the changes in gitmodules fv3atm hash: f595b972ab0a099a8ef13ddad313ec2c0d8672d4

Done

zach1221 · 2023-07-31T19:27:16Z

@DusanJovic-NOAA fv3atm PR #680 has been merged. Please update the sub-module pointer and revert the changes in gitmodules fv3atm hash: f595b972ab0a099a8ef13ddad313ec2c0d8672d4

Done

Thank you! Requests for final reviews have been sent.

DusanJovic-NOAA added 5 commits June 28, 2023 18:18

Update .gitmodules

59be99d

Update FV3

d7480b9

Update FV3

ab645bf

Merge remote-tracking branch 'origin/develop' into cubed_sphere_histo…

94d7ffb

…ry_output

Update FV3

ab2126e

DusanJovic-NOAA requested a review from junwang-noaa July 11, 2023 16:41

Update control_CubedSphereGrid_parallel test

cd00711

The control_CubedSphereGrid_parallel test has been updated to run post and create native grid history files

DusanJovic-NOAA added the Baseline Updates Current baselines will be updated. label Jul 12, 2023

junwang-noaa previously approved these changes Jul 12, 2023

View reviewed changes

zach1221 assigned FernandoAndrade-NOAA Jul 21, 2023

DusanJovic-NOAA added 2 commits July 25, 2023 18:30

Update FV3

8f71133

Merge remote-tracking branch 'origin/develop' into cubed_sphere_histo…

075583c

…ry_output

DusanJovic-NOAA dismissed junwang-noaa’s stale review via 075583c July 25, 2023 18:32

DusanJovic-NOAA added 2 commits July 27, 2023 17:45

Update FV3

1250495

Merge remote-tracking branch 'origin/develop' into cubed_sphere_histo…

1d80874

…ry_output

DusanJovic-NOAA added 2 commits July 28, 2023 15:33

Update FV3

014eb61

Merge remote-tracking branch 'origin/develop' into cubed_sphere_histo…

3fb0c02

…ry_output

FernandoAndrade-NOAA added Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. jenkins-ci Jenkins CI: ORT build/test on docker container labels Jul 28, 2023

FernandoAndrade-NOAA and others added 2 commits July 28, 2023 09:01

Update bl_date.conf

0974119

CubedSphereGrid test passes on WCOSS2, WCOSS2 RT Log

7f27783

add jet RT logs: passed

bcff14d

zach1221 added 3 commits July 31, 2023 12:36

add hera RT logs: passed

d60f3df

add orion RT logs: passed

c92d5b7

add gaea RT logs: passed

ed7bf6b

Revert .gitmodules and update FV3

c35f425

zach1221 requested review from SadeghTabas-NOAA, jkbk2004, DeniseWorthen and BrianCurtis-NOAA July 31, 2023 19:26

SadeghTabas-NOAA approved these changes Jul 31, 2023

View reviewed changes

BrianCurtis-NOAA approved these changes Jul 31, 2023

View reviewed changes

zach1221 merged commit 2d09adf into ufs-community:develop Jul 31, 2023

DusanJovic-NOAA deleted the cubed_sphere_history_output branch August 31, 2023 20:42

grantfirl mentioned this pull request Nov 2, 2023

UFS-dev PR#69 NCAR/ufs-weather-model#107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use inline post with cubed sphere history output #1831

Use inline post with cubed sphere history output #1831

DusanJovic-NOAA commented Jul 11, 2023 •

edited by zach1221

Loading

DusanJovic-NOAA commented Jul 11, 2023

junwang-noaa commented Jul 11, 2023

DusanJovic-NOAA commented Jul 11, 2023

junwang-noaa commented Jul 11, 2023

DusanJovic-NOAA commented Jul 11, 2023

FernandoAndrade-NOAA commented Jul 25, 2023

DusanJovic-NOAA commented Jul 25, 2023

FernandoAndrade-NOAA commented Jul 25, 2023

FernandoAndrade-NOAA commented Jul 26, 2023

FernandoAndrade-NOAA commented Jul 28, 2023 •

edited by zach1221

Loading

FernandoAndrade-NOAA commented Jul 28, 2023

FernandoAndrade-NOAA commented Jul 28, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

DusanJovic-NOAA commented Jul 31, 2023

zach1221 commented Jul 31, 2023

Use inline post with cubed sphere history output #1831

Use inline post with cubed sphere history output #1831

Conversation

DusanJovic-NOAA commented Jul 11, 2023 • edited by zach1221 Loading

Description

Input data additions/changes

Anticipated changes to regression tests:

Subcomponents involved:

Library Updates/Changes

Combined with PR's (If Applicable):

Commit Queue Checklist:

Linked PR's and Issues:

Testing Day Checklist:

Testing Log (for CM's):

DusanJovic-NOAA commented Jul 11, 2023

junwang-noaa commented Jul 11, 2023

DusanJovic-NOAA commented Jul 11, 2023

junwang-noaa commented Jul 11, 2023

DusanJovic-NOAA commented Jul 11, 2023

FernandoAndrade-NOAA commented Jul 25, 2023

DusanJovic-NOAA commented Jul 25, 2023

FernandoAndrade-NOAA commented Jul 25, 2023

FernandoAndrade-NOAA commented Jul 26, 2023

FernandoAndrade-NOAA commented Jul 28, 2023 • edited by zach1221 Loading

FernandoAndrade-NOAA commented Jul 28, 2023

FernandoAndrade-NOAA commented Jul 28, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

zach1221 commented Jul 31, 2023

DusanJovic-NOAA commented Jul 31, 2023

zach1221 commented Jul 31, 2023

DusanJovic-NOAA commented Jul 11, 2023 •

edited by zach1221

Loading

FernandoAndrade-NOAA commented Jul 28, 2023 •

edited by zach1221

Loading