Enable GOCART integration in UFS #310

rmontuoro · 2021-05-24T07:07:19Z

Description

This PR adds the required infrastructure to couple with UFS-Aerosols, a prognostic aerosol model based on NASA GOCART (see #309 and UFS weather model issue #590).

It also includes a hotfix to prevent segfault errors when coupling with chemistry (see PR #308)

Features:

Implemented sharing of atmospheric tracer metadata (name, units) to coupled model component via a standard NUOPC connector if coupling tracer field is accessed by shared memory reference
Introduced additional metadata to tracer records in field_table to identify and validate prognostic and diagnostic chemistry tracers. A new record line starting with the tracer_usage keyword is now required to recognize chemistry tracer as follows:

      "tracer_usage", "chemistry"                    (default, prognostic)
      "tracer_usage", "chemistry", "type=diagnostic" (diagnostic)

Added sanity check for input chemistry tracers, ensuring that both prognostic and diagnostic tracers in field_table are contiguous, and diagnostic tracers follow prognostic ones.
Improved algorithm to set aerosol scavenging factors
Exported instantaneous fields for:
- 3D non-convective liquid and ice precipitation fluxes
- 3D cloud fraction
- normalized soil wetness
- ice fraction (as used in the atmospheric model)
- lake fraction
- ocean fraction
- surface snow area fraction
Enabled import of diagnostic tracers

Fixes:

Prevent a divide by zero error when costant arrays are rescaled for lossy compression in write_netcdf_parallel()
Correctly sample and compute elapsed time for field bundle regridding operations.
Include hotfix (see PR Fix segfault errors in coupled FV3-GOCART after PR #301 #308) to allow latest setup_exportdata() to work with coupled chemistry.

This PR also eliminates legacy code used for GEFS-Aerosols coupling and no longer needed.

Issue(s) addressed

addresses Enable coupling with GOCART component #309
fixes Coupled FV3-GOCART system crashes after PR #301 #307

Testing

Successful regression tests performed for:

hera.intel
orion.intel
wcoss_dell_p3

Log files available at: https://github.com/rmontuoro/ufs-weather-model/tree/feature/gocart

This PR does not change the regression test baseline

Dependencies

waiting on Enable UFS-GOCART integration NCAR/ccpp-physics#664

large scale rain, and convective rain at the end of each coupling step if coupling with chemistry model.

defining zero and one.

in noah/osu land-surface model subdriver.

band layer cloud optical depths (0.55 and 10 mu channels) to prevent floating invalid errors due to uninitialized optical depth arrays.

coupling array at the beginning of each coupling step if coupled with chemistry model.

the NUOPC Realize phase since it breaks coupling with aerosol component.

This reverts commit 735eb9e.

via memory reference by default, if requested.

for timing of field bundle regridding operations.

coupled aerosol model if unavailable.

them to file to prevent dividing by zero.

to prevent segmentation fault.

Output cplgocart value.

coupled aerosol component (GEFS-Aerosols, GOCART).

setup_exportdata() to handle such return codes.

temperature to coupled chemistry model.

…egration

rmontuoro · 2021-06-03T19:35:46Z

The latest updates implement option 2a and enforce pointer association as agreed. @DusanJovic-NOAA, @DeniseWorthen , and @junwang-noaa please let me know if this works for you

DusanJovic-NOAA · 2021-06-03T19:57:38Z

atmos_model.F90

+            case default
+              localrc = ESMF_SUCCESS


The default case should be an error, not success.

@DusanJovic-NOAA - setup_exportdata() is not comprehensive. For instance, exported fields used by the chemistry model are not included in this subroutine and should not be at this time since they are exported at the end of FV3 run phase 1.
setup_exportdata() is used during FV3 initialization and by update_atmos_model_state(), which is called in run phase 2, except when FV3 is coupled with chemistry.

I understand the motivation, but setting the default case to be an error would break the coupled model.
Again, refactoring this code should not be part of this PR.

setup_exportdata is not used if cplchem is true, so the fact that not all chemistry fields are exported is irrelevant.

Why would it break the coupled model? Are there still exported fields (other than the 'chemistry' fields) that are not handled by the 'select case'?

setup_exportdata() is called by InitializeRealize() (fv3_cap.F90) even if cplchm is .true., and this is where all those issues are coming from.

If setup_exportdata() was not intended to be used with the coupled chemistry model, I'd be happy to disable it when cplchm is set to .true.. Please let me know.

I do not know if you need to call setup_exportdata with the chemistry model from InitializeRealize or not. But if you do then add all exported fields, and set the default case to error. If the default case is not an error there's no way to catch typos. For example, if instead:
case ('o3mr')
I write:
case ('o3nr')
model will silently continue to run, without me noticing that o3mr is not exported.

setup_exportdata() is not required at all when coupling with the chemistry model. I do agree with you that, generally speaking, we should return an error when a mismatch occurs.

I suggest:

checking if cplchm is .true. inside setup_exportdata() and bailing out if so (this is needed since GFS_control%cplchm is not accessible in the FV3 cap)

returning with an error in the default case, as you suggest.

Would this work?

I think it should.

@DusanJovic-NOAA - I've implemented the changes above. Please take a look.

exported ESMF field in setup_exportdata().

junwang-noaa · 2021-06-05T12:48:16Z

I replied in email, but I was told there is a temporary issue with github. I reposted the message here.

Raffaele, I think we discussed this at the last fv3-jedi meeting, we should use one single interface setup_exportdata for all the applications including fv3-mom6/cice6/ww3, fv3-jedi and fv3-chem. If there are fields used by several applications with the same names but associated with different fv3 internal fields, we can use namelist variables (e.g. cplchm) to set up the correct field value. Also the setup_exportdata can be called at InitializeRealize() if the internal coupled fields are properly initialized. It will be called again at the end of fv3 forecast phase1, so you will get updated fv3 export fields before forecast phase2 is called, I don't think there is any issue there. Using both setup_exportdata and update_atmos_chemistry for setting up fv3 export fields are causing confusion. @rsdunlapiv @@theurich @mark-a-potts can comment on this.

…ysics

climbfuji · 2021-06-30T22:26:21Z

I updated the ccpp-physics submodule pointer following the merge of NCAR/ccpp-physics#664 and reverted the change to .gitmodules. Please check @DusanJovic-NOAA @junwang-noaa @rmontuoro

DusanJovic-NOAA

Looks good.

junwang-noaa · 2021-07-01T01:46:19Z

atmos_model.F90

+              call block_data_copy_or_fill(datar82d, DYCORE_data(nb)%coupling%z_bot, zeror8, Atm_block, nb, rc=localrc)
+            !--- JEDI fields
+            case ('u')
+              call block_atmos_copy(datar83d, Atm(mygrid)%u, Atm_block, nb, rc=localrc)


Do you run fv-gocart test with 64bit? the Atm(mygrid)%u can be real(4) is compiled with 32BIT=Y.

This block is not used when coupling with GOCART.

We may have issue for this with 32BIT=Y, but this can be fixed later.

junwang-noaa · 2021-07-01T01:57:47Z

io/module_write_netcdf_parallel.F90

-              deallocate(arrayr4_3d_save)
-              call mpi_allreduce(mpi_in_place,compress_err(i),1,mpi_real4,mpi_max,mpi_comm,ierr)
-              !print *,'field name=',trim(fldName),dataMin,dataMax,compress_err(i)
+              if (scale_fact > 0.) then


Is original code causing issues or the code changes are to skip the allocating array and mpi calls for constant field?

This change is meant to address an issue in the original code.

Do you know what is the issue?

It was a divide by zero if scale_fact = 0

junwang-noaa

The code changes are working. Please see some questions of mine.

rmontuoro added 30 commits December 3, 2019 23:42

Reset to zero coupling arrays for accumulated snow,

f78aa42

large scale rain, and convective rain at the end of each coupling step if coupling with chemistry model.

Properly set kind type of literal constants

75654ac

defining zero and one.

Initialize to zero canopy resistance output variable

2870f5e

in noah/osu land-surface model subdriver.

Re-implement radiation diagnostic output involving spectral

37fb245

band layer cloud optical depths (0.55 and 10 mu channels) to prevent floating invalid errors due to uninitialized optical depth arrays.

Reset to zero instantaneous total moisture tendency

1c0a9ba

coupling array at the beginning of each coupling step if coupled with chemistry model.

Merge NOAA-EMC branch 'develop' into develop.

afb4c67

Merge NOAA-EMC branch 'develop' into develop.

3004022

Merge NOAA-EMC branch 'develop' into develop.

617b907

Merge EMC-NOAA branch 'develop' into develop.

bdd1131

Temporarily disable filling export fields during

735eb9e

the NUOPC Realize phase since it breaks coupling with aerosol component.

Moving previous commit to a branch.

59cf366

This reverts commit 735eb9e.

Merge NOAA-EMC branch 'develop' into develop

4849674

Allow coupled components to access exported fields

862f9e8

via memory reference by default, if requested.

Add missing time sampling call and rearrange time sampling

824d9d5

for timing of field bundle regridding operations.

Implement preliminary coupling to NASA GOCART.

a6d0ebd

Prevent importing diagnostic quantities from

be9bf3a

coupled aerosol model if unavailable.

Do not rescale constant arrays before writing

fb489d4

them to file to prevent dividing by zero.

Use proper CCPP control data type.

ed69d2c

Enable diagnostic tracers for aerosols.

781d718

Update ccpp/physics submodule to point to forked repository.

61dccf3

Fill export fields only if data array is allocated

c9daf4a

to prevent segmentation fault.

Update ccpp/physics submodule.

b13d76c

Add metadata for logical flag 'cplgocart'.

1912e9e

Do not set cplflx to .true. when coupling with GOCART.

94967f8

Output cplgocart value.

Populate only the export fields required by each

ebc4fec

coupled aerosol component (GEFS-Aerosols, GOCART).

Merge NOAA-EMC branch 'develop' into develop

0619729

Merge branch 'develop' into feature/gocart-integration

6e95699

Update atmos_cubed_sphere submodule to point to fork.

c7f26f0

Export instantaneous normalized soil wetness.

abb3134

Export instantaneous 3D cloud fraction.

336ad7f

rmontuoro added 6 commits June 3, 2021 03:43

Use proper datatype to retrieve instantaneous land/sea/ice mask.

be17583

Implement return code in block data methods and update

783321a

setup_exportdata() to handle such return codes.

Update error message.

911b4c4

Initialize return code variable.

9d29ae8

Use coupling data array to export instantaneous surface

520418b

temperature to coupled chemistry model.

Merge branch 'hotfix/NOAA-EMC#301-exportdata' into feature/gocart-int…

24cb285

…egration

rmontuoro requested a review from DusanJovic-NOAA June 3, 2021 19:31

DusanJovic-NOAA reviewed Jun 3, 2021

View reviewed changes

Return with error if no match is found for an

0aafc8f

exported ESMF field in setup_exportdata().

rmontuoro requested a review from DusanJovic-NOAA June 3, 2021 20:50

Disable setup_exportdata() if coupling with chemistry.

88a1be6

rmontuoro and others added 8 commits June 13, 2021 18:12

Merge branch 'develop' into feature/gocart-integration

77550ca

Merge NOAA-EMC branch 'develop' into feature/gocart-integration

02314a0

Update ccpp/physics submodule.

b2c5d23

Merge branch 'develop' into feature/gocart-integration

96960c4

Update submodule pointer for ccpp-physics

aeab36a

Update submodule pointer for ccpp-physics

b28c9de

Update submodule pointer for ccpp-physics

ef6ba4c

Revert change to .gitmodules and update submodule pointer for ccpp-ph…

86ae84d

…ysics

climbfuji requested a review from junwang-noaa June 30, 2021 22:25

DusanJovic-NOAA approved these changes Jun 30, 2021

View reviewed changes

junwang-noaa reviewed Jul 1, 2021

View reviewed changes

junwang-noaa approved these changes Jul 1, 2021

View reviewed changes

DusanJovic-NOAA merged commit ad7bddc into NOAA-EMC:develop Jul 1, 2021

grantfirl mentioned this pull request Jul 1, 2021

update GFS_typedefs.F90/meta to work with latest main ccpp-physics NCAR/ccpp-scm#268

Merged

rmontuoro mentioned this pull request Aug 23, 2021

Ensure tracer mass conservation during preconditioning NOAA-GFDL/GFDL_atmos_cubed_sphere#126

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable GOCART integration in UFS #310

Enable GOCART integration in UFS #310

rmontuoro commented May 24, 2021 •

edited

Loading

rmontuoro commented Jun 3, 2021

DusanJovic-NOAA Jun 3, 2021

rmontuoro Jun 3, 2021 •

edited

Loading

DusanJovic-NOAA Jun 3, 2021

rmontuoro Jun 3, 2021

DusanJovic-NOAA Jun 3, 2021

rmontuoro Jun 3, 2021

DusanJovic-NOAA Jun 3, 2021

rmontuoro Jun 4, 2021

junwang-noaa commented Jun 5, 2021

climbfuji commented Jun 30, 2021

DusanJovic-NOAA left a comment

junwang-noaa Jul 1, 2021

rmontuoro Jul 1, 2021

junwang-noaa Jul 1, 2021

junwang-noaa Jul 1, 2021

rmontuoro Jul 1, 2021

junwang-noaa Jul 1, 2021

rmontuoro Jul 1, 2021

junwang-noaa left a comment

Enable GOCART integration in UFS #310

Enable GOCART integration in UFS #310

Conversation

rmontuoro commented May 24, 2021 • edited Loading

Description

Issue(s) addressed

Testing

Dependencies

rmontuoro commented Jun 3, 2021

Choose a reason for hiding this comment

rmontuoro Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junwang-noaa commented Jun 5, 2021

climbfuji commented Jun 30, 2021

DusanJovic-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junwang-noaa left a comment

Choose a reason for hiding this comment

rmontuoro commented May 24, 2021 •

edited

Loading

rmontuoro Jun 3, 2021 •

edited

Loading