Turn on HAFS RTs on WCOSS2; AQM radians to degrees; NRL Neptune model 32-bit physics support #1227

BinLiu-NOAA · 2022-05-20T13:05:54Z

PR Checklist

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.
- Need at least create new baseline for the HAFS RTs on WCOSS2
- Need hafs_regional_storm_following_1nest_atm on hera.intel
New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsibility to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?

Turn on HAFS RTs on WCOSS2 and switch to use the netcdf (instead of netcdf_parallel) write_grid_component option for HAFS RTs.
- When using the netcdf_parallel write grid component option, the forecast will hang on WCOSS2 during writing out netcdf history output files. There were similar issues on other platforms occasionally as well.
Update to use fms/2022.01 on wcoss_cray.
This PR combines PR AQM radians to degrees #1212
- To fix an offset of PM2.5 peaks a coordinate change from radians to degrees was made.
- There are no changes to results as there are no RT's for AQM yet.
- Recommend bringing this PR in before AQM RT addition or along with it.
This PR also combines PR NRL Neptune model 32-bit physics support #1206
- Adds ccpp-physics changes needed to support 32-bit physics on Neptune. This does not add 32-bit physics to FV3, which requires quite a lot more changes in several repositories.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.
(Remember, issues must always be created before starting work on a PR branch!)

fixes Turn on HAFS related regression tests on WCOSS2 #1226
fixes Update to use fms/2022.01 on wcoss_cray #1229
potentially fix HAFS nested tests failed on several platforms #1085
fixes Bring in AQM changes to fix issue with PM2.5 peaks being offset #1213
Implements what this PR was supposed to:
- NRL single-precision changes NCAR/ccpp-physics#797
Which is about half of this batch of changes from NRL:
- Single precision suite-4 from NEPTUNE -- branch jm-nrl-32bitfp-24cc09e -> main NCAR/ccpp-physics#772

Testing

Tests conducted on WCOSS2 run through all HAFS RTs.

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

Dependencies

If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).

waiting on update of aqm_cap for coordinate change NOAA-EMC/AQM#2
waiting on NRL Neptune model 32-bit physics support NCAR/ccpp-physics#918
waiting on NRL Neptune model 32-bit physics support NOAA-EMC/fv3atm#532

netcdf_parallel) write_grid_component option for HAFS RTs.

…weather-model into ccpp-neptune

…hafs_wcoss2_rts

tests/rt.conf

BrianCurtis-NOAA · 2022-05-25T17:50:29Z

@BinLiu-NOAA Please add a WCOSS2 item in the ufs-weather-model/.github/pull_request_template.md

…ra.intel.

BinLiu-NOAA · 2022-05-25T18:27:01Z

@BinLiu-NOAA Please add a WCOSS2 item in the ufs-weather-model/.github/pull_request_template.md

@BrianCurtis-NOAA, done. Thanks!

…ss2_rts

BrianCurtis-NOAA · 2022-05-25T18:56:55Z

@BinLiu-NOAA Its now your turn at the top of the commit queue. Please update .gitmodules with #1206 (fv3atm) changes and #1212 (AQM) changes.

on-behalf-of @ufs-community <brian.curtis@noaa.gov>

BrianCurtis-NOAA · 2022-05-26T12:56:47Z

@BinLiu-NOAA With the error of variables being different in the hafs_regional_.._gfdl test, you're OK with regenerating that one test, correct?

BinLiu-NOAA · 2022-05-26T13:02:51Z

@BinLiu-NOAA With the error of variables being different in the hafs_regional_.._gfdl test, you're OK with regenerating that one test, correct?

Yes, please help to go ahead regenerating baseline for this test. There is a change from netcdf_parallel to netcdf output option, which might have affected the netcdf output files. Thanks!

junwang-noaa · 2022-05-26T13:08:01Z

@BinLiu-NOAA Would you please check the differences between the two files(netcdf_parallel and netcdf) to confirm the data fields are correct? Thanks

BinLiu-NOAA · 2022-05-26T13:54:28Z

@BinLiu-NOAA Would you please check the differences between the two files(netcdf_parallel and netcdf) to confirm the data fields are correct? Thanks

@junwang-noaa and @BrianCurtis-NOAA, Sure, doing that. Actually found an interesting issue when comparing the new generated baseline:
ncdump -h /gpfs/hps3/emc/nems/noscrub/emc.nemspara/RT/NEMSfv3gfs/develop-20220519/hafs_regional_atm_thompson_gfdlsf/atmf006.nc
against the previous version baseline on wcoss_cray:
ncdump -h /gpfs/hps3/emc/nems/noscrub/emc.nemspara/RT/NEMSfv3gfs/develop-20220516/hafs_regional_atm_thompson_gfdlsf/atmf006.nc

The new baseline got the following three variables missing, for which I don't know why. Other than that the other variables actually got identical values. I believe the only thing this PR changed for this hafs_regional_atm_thompson_gfdlsf test is using netcdf instead of netcdf_parallel output. Also, another special thing for this test is it uses the "cubed_sphere" native grid output option.

    char time_iso(time, nchars) ;
            time_iso:long_name = "valid time" ;
            time_iso:description = "ISO 8601 datetime string" ;
    double lon(grid_yt, grid_xt) ;
            lon:long_name = "T-cell longitude" ;
            lon:units = "degrees_E" ;
    double lat(grid_yt, grid_xt) ;
            lat:long_name = "T-cell latitude" ;
            lat:units = "degrees_N" ;

SamuelTrahanNOAA · 2022-05-26T14:45:03Z

The fv3gfs_diag_output never sends those three variables directly, and they're not in the diag_table. Maybe the diag manager is supposed to add them automatically? If so, this is a bug in FMS, beyond Bin's control.

Edit: They're not in any diag tables in tests/parm/diag_table

DusanJovic-NOAA · 2022-05-26T14:45:43Z

@BinLiu-NOAA Would you please check the differences between the two files(netcdf_parallel and netcdf) to confirm the data fields are correct? Thanks

@junwang-noaa and @BrianCurtis-NOAA, Sure, doing that. Actually found an interesting issue when comparing the new generated baseline: ncdump -h /gpfs/hps3/emc/nems/noscrub/emc.nemspara/RT/NEMSfv3gfs/develop-20220519/hafs_regional_atm_thompson_gfdlsf/atmf006.nc against the previous version baseline on wcoss_cray: ncdump -h /gpfs/hps3/emc/nems/noscrub/emc.nemspara/RT/NEMSfv3gfs/develop-20220516/hafs_regional_atm_thompson_gfdlsf/atmf006.nc

The new baseline got the following three variables missing, for which I don't know why. Other than that the other variables actually got identical values. I believe the only thing this PR changed for this hafs_regional_atm_thompson_gfdlsf test is using netcdf instead of netcdf_parallel output. Also, another special thing for this test is it uses the "cubed_sphere" native grid output option.
    char time_iso(time, nchars) ;
            time_iso:long_name = "valid time" ;
            time_iso:description = "ISO 8601 datetime string" ;
    double lon(grid_yt, grid_xt) ;
            lon:long_name = "T-cell longitude" ;
            lon:units = "degrees_E" ;
    double lat(grid_yt, grid_xt) ;
            lat:long_name = "T-cell latitude" ;
            lat:units = "degrees_N" ;

Different output routines are used for 'cubed_sphere_grid' output_grid with 'netcdf' (ESMFproto_FieldBundleWrite) than for 'netcdf_parallel' (write_netcdf).

junwang-noaa · 2022-05-26T14:48:47Z

@BinLiu-NOAA Thanks for checking. I think the valid time was only added to non-native grid history file output. We are using ESMF calls to write out native grid output. This is a different issue not related to this PR. @DusanJovic-NOAA FYI.

BinLiu-NOAA · 2022-05-26T14:51:12Z

Different output routines are used for 'cubed_sphere_grid' output_grid with 'netcdf' (ESMFproto_FieldBundleWrite) than for 'netcdf_parallel' (write_netcdf).

Thanks for this information, @DusanJovic-NOAA! With that, @junwang-noaa and @BrianCurtis-NOAA, I would suggest just moving forward with the new re-generated baseline for this specific hafs_regional_atm_thompson_gfdlsf test. And let this issue/inconsistency be fixed/addressed in a future PR. Thanks!

BrianCurtis-NOAA · 2022-05-26T18:22:16Z

Automated RT Failure Notification
Machine: gaea
Compiler: intel
Job: RT
[RT] Repo location: /lustre/f2/pdata/ncep/emc.nemspara/autort/pr/942703315/20220526113015/ufs-weather-model
[RT] Error: Test hafs_regional_atm_thompson_gfdlsf 095 failed in check_result failed
[RT] Error: Test hafs_regional_atm_thompson_gfdlsf 095 failed in run_test failed
Please make changes and add the following label back: gaea-intel-RT

junwang-noaa · 2022-05-26T20:13:09Z

@BrianCurtis-NOAA Are you going to rerun the hafs_regional_atm_thompson_gfdlsf test?

BrianCurtis-NOAA · 2022-05-26T20:15:00Z

@BrianCurtis-NOAA Are you going to rerun the hafs_regional_atm_thompson_gfdlsf test?

Yes I am. @DeniseWorthen did a compare and found new/previous baseline to be the same even though it showed as failed. I started the re-run before 3 but it's stuck in the Gaea queue still. Apologies it's going slow today.

BrianCurtis-NOAA · 2022-05-26T20:27:45Z

@junwang-noaa ready for merges for ccpp-physics and AQM

BrianCurtis-NOAA · 2022-05-26T20:45:13Z

@BinLiu-NOAA Please update .gitmodules and AQM 73da18b and FV3 515a39d hashes

SamuelTrahanNOAA and others added 7 commits May 6, 2022 20:44

Changes to ccpp to support 32-bit physics in NRL Neptune model

4e030a6

Update .gitmodules

f36ed12

hera.gnu tests passed

a3d5551

hera.intel tests passed

1747c0f

jet.intel tests passed

a808903

bring in AQM to ufs-model

90e87b7

Turn on HAFS RTs on WCOSS2 and switch to use the netcdf (instead of

9d8f885

netcdf_parallel) write_grid_component option for HAFS RTs.

BinLiu-NOAA added enhancement New feature or request Baseline Updates Current baselines will be updated. labels May 20, 2022

This was referenced May 20, 2022

Turn on HAFS related regression tests on WCOSS2 #1226

Closed

Update to use fms/2022.01 on wcoss_cray #1229

Closed

SamuelTrahanNOAA and others added 8 commits May 23, 2022 18:56

merge develop

475289c

hera gnu logs

67aa808

jet intel tests passed

40e021f

Merge branch 'ccpp-neptune' of ssh://github.com/SamuelTrahanNOAA/ufs-…

6fa50f1

…weather-model into ccpp-neptune

change ssh -> https

80f7469

FV3/ccpp/physics: Simplify machine.F and remove unused types.

510059b

Merge remote-tracking branch 'remotes/upstream/develop' into feature/…

50ec7f7

…hafs_wcoss2_rts

Use fms/2022.01 on wcoss_cray.

ba50334

DeniseWorthen reviewed May 25, 2022

View reviewed changes

tests/rt.conf Outdated Show resolved Hide resolved

BinLiu-NOAA added 2 commits May 25, 2022 13:04

Turn this hafs_regional_storm_following_1nest_atm test back on for he…

6ed2b4c

…ra.intel.

Add the wcoss2.intel platform in pull_request_template.md.

68e05fd

DeniseWorthen linked an issue May 25, 2022 that may be closed by this pull request

HAFS nested tests failed on several platforms #1085

Closed

SamuelTrahanNOAA and others added 2 commits May 25, 2022 18:42

merge develop

4bfa00b

Merge remote-tracking branch 'upstream/develop' into feature/hafs_wco…

251ddc0

…ss2_rts

This was referenced May 25, 2022

NRL Neptune model 32-bit physics support #1206

Closed

AQM radians to degrees #1212

Closed

BrianCurtis-NOAA added 5 commits May 26, 2022 02:36

[AutoRT] hera.intel Job Completed.

d28899d

on-behalf-of @ufs-community <brian.curtis@noaa.gov>

[AutoRT] jet.intel Job Completed.

8936239

on-behalf-of @ufs-community <brian.curtis@noaa.gov>

[AutoRT] hera.gnu Job Completed.

844df28

on-behalf-of @ufs-community <brian.curtis@noaa.gov>

[AutoRT] cheyenne.intel Job Completed.

559f6ad

on-behalf-of @ufs-community <brian.curtis@noaa.gov>

WCOSS Dell P3 RT Passed

0232cfb

BrianCurtis-NOAA added the run-ci label May 26, 2022

github-actions bot removed the run-ci label May 26, 2022

BrianCurtis-NOAA added gaea-intel-RT and removed gaea-intel-RT labels May 26, 2022

junwang-noaa requested review from DusanJovic-NOAA and MinsukJi-NOAA May 26, 2022 12:24

Gaea.intel log passed

8b5d80c

DusanJovic-NOAA approved these changes May 26, 2022

View reviewed changes

MinsukJi-NOAA approved these changes May 26, 2022

View reviewed changes

revert .gitmodules and update AQM and FV3 hashes

aa849a9

BrianCurtis-NOAA merged commit 71c752a into ufs-community:develop May 26, 2022

BinLiu-NOAA deleted the feature/hafs_wcoss2_rts branch May 26, 2022 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Turn on HAFS RTs on WCOSS2; AQM radians to degrees; NRL Neptune model 32-bit physics support #1227

Turn on HAFS RTs on WCOSS2; AQM radians to degrees; NRL Neptune model 32-bit physics support #1227

BinLiu-NOAA commented May 20, 2022 •

edited by BrianCurtis-NOAA

Loading

BrianCurtis-NOAA commented May 25, 2022

BinLiu-NOAA commented May 25, 2022

BrianCurtis-NOAA commented May 25, 2022

BrianCurtis-NOAA commented May 26, 2022

BinLiu-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BinLiu-NOAA commented May 26, 2022 •

edited

Loading

SamuelTrahanNOAA commented May 26, 2022 •

edited

Loading

DusanJovic-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BinLiu-NOAA commented May 26, 2022

BrianCurtis-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BrianCurtis-NOAA commented May 26, 2022 •

edited

Loading

BrianCurtis-NOAA commented May 26, 2022 •

edited

Loading

BrianCurtis-NOAA commented May 26, 2022

Turn on HAFS RTs on WCOSS2; AQM radians to degrees; NRL Neptune model 32-bit physics support #1227

Turn on HAFS RTs on WCOSS2; AQM radians to degrees; NRL Neptune model 32-bit physics support #1227

Conversation

BinLiu-NOAA commented May 20, 2022 • edited by BrianCurtis-NOAA Loading

PR Checklist

Instructions: All subsequent sections of text should be filled in as appropriate.

Description

Issue(s) addressed

Testing

Dependencies

BrianCurtis-NOAA commented May 25, 2022

BinLiu-NOAA commented May 25, 2022

BrianCurtis-NOAA commented May 25, 2022

BrianCurtis-NOAA commented May 26, 2022

BinLiu-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BinLiu-NOAA commented May 26, 2022 • edited Loading

SamuelTrahanNOAA commented May 26, 2022 • edited Loading

DusanJovic-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BinLiu-NOAA commented May 26, 2022

BrianCurtis-NOAA commented May 26, 2022

junwang-noaa commented May 26, 2022

BrianCurtis-NOAA commented May 26, 2022 • edited Loading

BrianCurtis-NOAA commented May 26, 2022 • edited Loading

BrianCurtis-NOAA commented May 26, 2022

BinLiu-NOAA commented May 20, 2022 •

edited by BrianCurtis-NOAA

Loading

BinLiu-NOAA commented May 26, 2022 •

edited

Loading

SamuelTrahanNOAA commented May 26, 2022 •

edited

Loading

BrianCurtis-NOAA commented May 26, 2022 •

edited

Loading

BrianCurtis-NOAA commented May 26, 2022 •

edited

Loading