Skip to content

Commit

Permalink
Merge pull request #506 from billsacks/fix_xgrid_reproducibility
Browse files Browse the repository at this point in the history
Fix xgrid reproducibility by using srcTermProcessing=0 for all xgrid FieldRegridStore calls

### Description of changes

Use srcTermProcessing=0 for all xgrid FieldRegridStore calls.

This is needed for bit-for-bit reproducibility. Most of the calls to FieldRegridStore weren't setting srcTermProcessing at all; this can lead to irreproducibility of the results. Two of the calls had srcTermProcessing set to 1 before, which leads to reproducibility but (according to Gerhard Theurich) is often worse for performance than a value of 0.

Note that we use a value of 0 in the calls in med_map_mod.

### Specific notes

Contributors other than yourself, if any: Guidance from @oehmke and @theurich

CMEPS Issues Fixed (include github issue #): Resolves #505 

Are changes expected to change answers? YES: roundoff-level differences expected for runs with `aoflux_grid = "xgrid"` (but those runs had expected roundoff-level differences from run to run before this fix anyway)

Any User Interface Changes (namelist or namelist defaults changes)? No

### Testing performed
Please describe the tests along with the target model and machine(s) 
If possible, please also added hashes that were used in the testing

In the context of cesm3_0_alpha03c, with the change here put on top of the change in #501, ran `REP_Ld2.ne30pg3_t232.BLT1850.derecho_intel.allactive-xgrid`, where the `xgrid` testmod contained:

include_user_mods:
```
../defaultio
```

user_nl_cpl:
```
aoflux_grid = "xgrid"
```
  • Loading branch information
billsacks authored Sep 28, 2024
2 parents b1e2325 + 17b48d9 commit 842be8a
Showing 1 changed file with 17 additions and 10 deletions.
27 changes: 17 additions & 10 deletions mediator/med_phases_aofluxes_mod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -780,14 +780,16 @@ subroutine med_aofluxes_init_xgrid(gcomp, aoflux_in, aoflux_out, rc)
type(ESMF_Mesh) :: xch_mesh
real(r8), pointer :: dataptr(:)
integer :: fieldcount
integer :: stp ! srcTermProcessing is declared inout and must have variable not constant
integer :: srcTermProcessing_Value ! srcTermProcessing is declared inout and must have variable not constant
type(ESMF_CoordSys_Flag) :: coordSys
real(ESMF_KIND_R8) ,allocatable :: garea(:)
character(len=*),parameter :: subname=' (med_aofluxes_init_xgrid) '
!-----------------------------------------------------------------------

rc = ESMF_SUCCESS

srcTermProcessing_Value = 0

! Get the internal state from the mediator Component.
nullify(is_local%wrap)
call ESMF_GridCompGetInternalState(gcomp, is_local, rc)
Expand Down Expand Up @@ -877,23 +879,26 @@ subroutine med_aofluxes_init_xgrid(gcomp, aoflux_in, aoflux_out, rc)
dataptr(:) = 1.0_r8

! create agrid->xgrid route handles
call ESMF_FieldRegridStore(xgrid, field_a, field_x, routehandle=rh_agrid2xgrid, rc=rc)
call ESMF_FieldRegridStore(xgrid, field_a, field_x, routehandle=rh_agrid2xgrid, &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldRegridStore(xgrid, field_a, field_x, routehandle=rh_agrid2xgrid_2ndord, &
regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND, rc=rc)
regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND, srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
if (trim(coupling_mode) == 'cesm') then
stp = 1
call ESMF_FieldRegridStore(field_a, field_x, routehandle=rh_agrid2xgrid_bilinr, &
regridmethod=ESMF_REGRIDMETHOD_BILINEAR, dstMaskValues=(/0/), srcTermProcessing=stp, rc=rc)
regridmethod=ESMF_REGRIDMETHOD_BILINEAR, dstMaskValues=(/0/), &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldRegridStore(field_a, field_x, routehandle=rh_agrid2xgrid_patch, &
regridmethod=ESMF_REGRIDMETHOD_PATCH, dstMaskValues=(/0/), srcTermProcessing=stp, rc=rc)
regridmethod=ESMF_REGRIDMETHOD_PATCH, dstMaskValues=(/0/), &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
end if

! create xgrid->zgrid route handle
call ESMF_FieldRegridStore(xgrid, field_x, field_a, routehandle=rh_xgrid2agrid, rc=rc)
call ESMF_FieldRegridStore(xgrid, field_x, field_a, routehandle=rh_xgrid2agrid, &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return

! destroy temporary field
Expand All @@ -911,12 +916,14 @@ subroutine med_aofluxes_init_xgrid(gcomp, aoflux_in, aoflux_out, rc)
call ESMF_FieldGet(field_o, farrayptr=dataptr, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
dataptr(:) = 1.0_r8
call ESMF_FieldRegridStore(xgrid, field_o, field_x, routehandle=rh_ogrid2xgrid, rc=rc)
call ESMF_FieldRegridStore(xgrid, field_o, field_x, routehandle=rh_ogrid2xgrid, &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldRegridStore(xgrid, field_x, field_o, routehandle=rh_xgrid2ogrid, rc=rc)
call ESMF_FieldRegridStore(xgrid, field_x, field_o, routehandle=rh_xgrid2ogrid, &
srcTermProcessing=srcTermProcessing_Value, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
! call ESMF_FieldRegridStore(xgrid, field_o, field_x, routehandle=rh_ogrid2xgrid_2ndord, &
! regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND, rc=rc)
! regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND, srcTermProcessing=srcTermProcessing_Value, rc=rc)
! if (chkerr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldDestroy(field_o, rc=rc)
if (chkerr(rc,__LINE__,u_FILE_u)) return
Expand Down

0 comments on commit 842be8a

Please sign in to comment.