Coupled run using benchmark at C384 crashed when frac_grid=T #268

ShanSunNOAA · 2020-11-10T06:25:40Z

Description

The coupled model runs well in the benchmark case at C384 with frac_grid=F & frac_grid_input=T. It crashed with frac_grid=T, on line 778 in module_gfdl_cloud_microphys.F90 where dz (i, j, k) becomes zero somewhere:
776 dz0 (k) = dz (i, j, k)
777
778 den0 (k) = - dp1 (k) / (grav * dz0 (k)) ! density of dry air

To Reproduce:

This can be reproduced in https://github.com/shansun6/ufs-weather-model, -b frac_bm_20201108. To run it, do
rt.sh -l rt.conf_bmark

Hera keeps frozen today, so I don't have output yet. When it is back to normal, I will add the output dir here.

yangfanglin · 2020-11-10T17:15:41Z

Shan, Can you repeat this run in debug mode to get more information ? How soon the model crashed ?

ShanSunNOAA · 2020-11-10T21:50:58Z

Hi Fanglin, Thanks for your email. The model crashed right away during the 1st time step. The error message of dz=0 on Line 778 in GFDL microphysics is from the debug mode. I will let you know when I have more info. Thanks, Shan

…

On Tue, Nov 10, 2020 at 10:16 AM Fanglin Yang ***@***.***> wrote: Shan, Can you repeat this run in debug mode to get more information ? How soon the model crashed ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALORMVUWYBEW4YWCFSR4GRDSPFYM3ANCNFSM4TQH56ZA> .

ShanSunNOAA · 2020-11-11T06:32:22Z

I made one modification today: slmsk=floor(landfrac) when frac_grid=T, to be consistent with ICs.

However, the model still crashed at the same place(Line 778 of module_gfdl_cloud_microphys.F90) during the 1st time step. The error using debug=Y is at /scratch2/BMC/gsd-fv3-dev/Shan.Sun/FV3_RT/rt_94187/cpld_bmark_frac_prod/ on hera.

junwang-noaa · 2020-11-11T14:59:35Z

Shan, the dz is computed from interface pressure phii in module_gfdl_cloud_microphys.F90. The phii is updated in get_phi_fv3, and it should not have same value at two consecutive layers unless the tmp (gt0) or there are two levels with same pressure in model physics state. I'd suggest to find the (i,j) location of dz=0 in dz(i,k) = (phii(i,kk)-phii(i,kk+1))*onebg in module_gfdl_cloud_microphys.F90, then check it in get_phi_fv3_run to see where tmp becomes 0.

ShanSunNOAA · 2020-11-11T17:16:10Z

Jun, thanks for your suggestion. I inserted a print statement after dz(i,k) = (phii(i,kk)-phii(i,kk+1))onebg in gfdl_cloud_microphys.F90:
if (abs(dz(i,k))<1.e-12) write(,'(a,2i4,a,2es10.2,2(a,es10.2))') 'warning1 dz=0 at i,k=',i,k,' phii=',phii(i,kk),phii(i,kk+1),' dz=',dz(i,k)

Here is the output: phii went bad at many points and different k:

109: warning1 dz=0 at i,k= 8 1 phii= 2.55E+71 2.55E+71 dz= 0.00E+00
109: warning1 dz=0 at i,k= 10 1 phii= 1.16E+73 1.16E+73 dz= 0.00E+00
109: warning1 dz=0 at i,k= 14 1 phii= 1.98E+72 1.98E+72 dz= 0.00E+00
109: warning1 dz=0 at i,k= 15 1 phii= 3.17E+72 3.17E+72 dz= 0.00E+00
109: warning1 dz=0 at i,k= 8 2 phii= 2.55E+71 2.55E+71 dz= 0.00E+00
109: warning1 dz=0 at i,k= 10 2 phii= 1.16E+73 1.16E+73 dz= 0.00E+00
109: warning1 dz=0 at i,k= 14 2 phii= 1.98E+72 1.98E+72 dz= 0.00E+00
109: warning1 dz=0 at i,k= 15 2 phii= 3.17E+72 3.17E+72 dz= 0.00E+00
109: warning1 dz=0 at i,k= 8 3 phii= 2.55E+71 2.55E+71 dz= 0.00E+00
109: warning1 dz=0 at i,k= 10 3 phii= 1.16E+73 1.16E+73 dz= 0.00E+00
. . .

see /scratch2/BMC/gsd-fv3-dev/Shan.Sun/FV3_RT/rt_73625/cpld_bmark_frac_prod/out

Also the error in "err" has switched to a different routine (it is no longer Line 778 of module_gfdl_cloud_microphys.F90):

139: forrtl: error (72): floating overflow
139: Image PC Routine Line Source
139: fv3.exe 000000000D25E6BF Unknown Unknown Unknown
139: libpthread-2.17.s 00002B7DB5698630 Unknown Unknown Unknown
139: fv3.exe 0000000005EE862F samfshalcnv_mp_sa 776 samfshalcnv.f

where Line 776 in samfshalcnv.f is the calculation of eta:
774 dz = zi(i,k) - zi(i,k-1)
775 ptem = 0.5*(xlamue(i,k)+xlamue(i,k-1))-xlamud(i)
776 eta(i,k) = eta(i,k-1) * (1 + ptem * dz)

Any suggestions where to go from here? Thanks,

SMoorthi-emc · 2020-11-11T19:56:46Z

This problem is related to a “huge” value sneaking in the fractional grid version, at least that was true in my case. I fixed it in my branch and I have no GFDL MP issue. I still have restart issue, so I am not pushing my PR yet. Moorthi

…

Sent from my iPhone

On Nov 11, 2020, at 12:16 PM, shansun6 ***@***.***> wrote: Jun, thanks for your suggestion. I inserted a print statement after dz(i,k) = (phii(i,kk)-phii(i,kk+1))onebg in gfdl_cloud_microphys.F90: if (abs(dz(i,k))<1.e-12) write(,'(a,2i4,a,2es10.2,2(a,es10.2))') 'warning1 dz=0 at i,k=',i,k,' phii=',phii(i,kk),phii(i,kk+1),' dz=',dz(i,k) Here is the output: phii went bad at many points and different k: 109: warning1 dz=0 at i,k= 8 1 phii= 2.55E+71 2.55E+71 dz= 0.00E+00 109: warning1 dz=0 at i,k= 10 1 phii= 1.16E+73 1.16E+73 dz= 0.00E+00 109: warning1 dz=0 at i,k= 14 1 phii= 1.98E+72 1.98E+72 dz= 0.00E+00 109: warning1 dz=0 at i,k= 15 1 phii= 3.17E+72 3.17E+72 dz= 0.00E+00 109: warning1 dz=0 at i,k= 8 2 phii= 2.55E+71 2.55E+71 dz= 0.00E+00 109: warning1 dz=0 at i,k= 10 2 phii= 1.16E+73 1.16E+73 dz= 0.00E+00 109: warning1 dz=0 at i,k= 14 2 phii= 1.98E+72 1.98E+72 dz= 0.00E+00 109: warning1 dz=0 at i,k= 15 2 phii= 3.17E+72 3.17E+72 dz= 0.00E+00 109: warning1 dz=0 at i,k= 8 3 phii= 2.55E+71 2.55E+71 dz= 0.00E+00 109: warning1 dz=0 at i,k= 10 3 phii= 1.16E+73 1.16E+73 dz= 0.00E+00 . . . see /scratch2/BMC/gsd-fv3-dev/Shan.Sun/FV3_RT/rt_73625/cpld_bmark_frac_prod/out Also the error in "err" has switched to a different routine (it is no longer Line 778 of module_gfdl_cloud_microphys.F90): 139: forrtl: error (72): floating overflow 139: Image PC Routine Line Source 139: fv3.exe 000000000D25E6BF Unknown Unknown Unknown 139: libpthread-2.17.s 00002B7DB5698630 Unknown Unknown Unknown 139: fv3.exe 0000000005EE862F samfshalcnv_mp_sa 776 samfshalcnv.f where Line 776 in samfshalcnv.f is the calculation of eta: 774 dz = zi(i,k) - zi(i,k-1) 775 ptem = 0.5*(xlamue(i,k)+xlamue(i,k-1))-xlamud(i) 776 eta(i,k) = eta(i,k-1) * (1 + ptem * dz) Any suggestions where to go from here? Thanks, — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

ShanSunNOAA · 2020-11-11T21:48:58Z

Thank you, Moorthi, for your info. Should I try your ccpp-physics branch SM/SM_Oct102020, or is there one routine that I can cherry pick? I just want to make the "frac+gfdl" run to complete first. Please advice. Thanks!

SMoorthi-emc · 2020-11-12T00:04:38Z

I think my branch does have the fix; I don't remember precisely what line. You can try my branch to see if it works for you. I have been running with GFDL MP and it works fine. The only problem I have is that restart is not reproducing - still trying to chase but CCPP is not that easy (everything works fine under IPD). Moorthi

…

On Wed, Nov 11, 2020 at 4:49 PM shansun6 ***@***.***> wrote: Thank you, Moorthi, for your info. Should I try your ccpp-physics branch SM/SM_Oct102020, or is there one routine that I can cherry pick? I just want to make the "frac+gfdl" run to complete first. Please advice. Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYXOBHCI6Q4XADNMHVDSPMBFTANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

SMoorthi-emc · 2020-11-12T00:50:28Z

Shan, My branch does things slightly different from what you are doing. I am using slmsk=1 if landfrac > 0.0. So, using my branch might cause some other issues for you. Moorthi On Wed, Nov 11, 2020 at 7:04 PM Shrinivas Moorthi - NOAA Federal < shrinivas.moorthi@noaa.gov> wrote:

…

I think my branch does have the fix; I don't remember precisely what line. You can try my branch to see if it works for you. I have been running with GFDL MP and it works fine. The only problem I have is that restart is not reproducing - still trying to chase but CCPP is not that easy (everything works fine under IPD). Moorthi On Wed, Nov 11, 2020 at 4:49 PM shansun6 ***@***.***> wrote: > Thank you, Moorthi, for your info. Should I try your ccpp-physics branch > SM/SM_Oct102020, or is there one routine that I can cherry pick? I just > want to make the "frac+gfdl" run to complete first. Please advice. Thanks! > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#268 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALLVRYXOBHCI6Q4XADNMHVDSPMBFTANCNFSM4TQH56ZA> > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

junwang-noaa · 2020-11-12T01:09:47Z

I guess Moorthi might make some changes in get_phi_fv3_run or the gt0 passed to get_phi_fv3_run. On Wed, Nov 11, 2020 at 7:50 PM SMoorthi-emc <notifications@github.com> wrote:

…

Shan, My branch does things slightly different from what you are doing. I am using slmsk=1 if landfrac > 0.0. So, using my branch might cause some other issues for you. Moorthi On Wed, Nov 11, 2020 at 7:04 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote: > I think my branch does have the fix; I don't remember precisely what line. > You can try my branch to see if it works for you. > I have been running with GFDL MP and it works fine. > The only problem I have is that restart is not reproducing - still trying > to chase but CCPP is not that easy (everything works fine under IPD). > Moorthi > > On Wed, Nov 11, 2020 at 4:49 PM shansun6 ***@***.***> wrote: > >> Thank you, Moorthi, for your info. Should I try your ccpp-physics branch >> SM/SM_Oct102020, or is there one routine that I can cherry pick? I just >> want to make the "frac+gfdl" run to complete first. Please advice. Thanks! >> >> — >> You are receiving this because you commented. >> Reply to this email directly, view it on GitHub >> < #268 (comment) >, >> or unsubscribe >> < https://github.com/notifications/unsubscribe-auth/ALLVRYXOBHCI6Q4XADNMHVDSPMBFTANCNFSM4TQH56ZA > >> . >> > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TIBV3QNU73VZK6ZWIDSPMWOBANCNFSM4TQH56ZA> .

SMoorthi-emc · 2020-11-12T01:27:15Z

No, I did not. The cause of the problem is in merging with "frac_grid=.true." I think I made changes in sfc_composit routine. The problem is that surface T (and probably q) was becoming too large. Moorthi

…

On Wed, Nov 11, 2020 at 8:10 PM Jun Wang ***@***.***> wrote: I guess Moorthi might make some changes in get_phi_fv3_run or the gt0 passed to get_phi_fv3_run. On Wed, Nov 11, 2020 at 7:50 PM SMoorthi-emc ***@***.***> wrote: > Shan, > My branch does things slightly different from what you are doing. I am > using slmsk=1 if landfrac > 0.0. > So, using my branch might cause some other issues for you. > Moorthi > > > On Wed, Nov 11, 2020 at 7:04 PM Shrinivas Moorthi - NOAA Federal < > ***@***.***> wrote: > > > I think my branch does have the fix; I don't remember precisely what > line. > > You can try my branch to see if it works for you. > > I have been running with GFDL MP and it works fine. > > The only problem I have is that restart is not reproducing - still trying > > to chase but CCPP is not that easy (everything works fine under IPD). > > Moorthi > > > > On Wed, Nov 11, 2020 at 4:49 PM shansun6 ***@***.***> > wrote: > > > >> Thank you, Moorthi, for your info. Should I try your ccpp-physics branch > >> SM/SM_Oct102020, or is there one routine that I can cherry pick? I just > >> want to make the "frac+gfdl" run to complete first. Please advice. > Thanks! > >> > >> — > >> You are receiving this because you commented. > >> Reply to this email directly, view it on GitHub > >> < > #268 (comment) > >, > >> or unsubscribe > >> < > https://github.com/notifications/unsubscribe-auth/ALLVRYXOBHCI6Q4XADNMHVDSPMBFTANCNFSM4TQH56ZA > > > >> . > >> > > > > > > -- > > Dr. Shrinivas Moorthi > > Research Meteorologist > > Modeling and Data Assimilation Branch > > Environmental Modeling Center / National Centers for Environmental > > Prediction > > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > > Tel: (301)683-3718 > > > > e-mail: ***@***.*** > > Phone: (301) 683-3718 Fax: (301) 683-3718 > > > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AI7D6TIBV3QNU73VZK6ZWIDSPMWOBANCNFSM4TQH56ZA > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYUQGU4OWQK2JDNN53LSPMYWPANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

ShanSunNOAA · 2020-11-12T03:29:35Z

Hi Moorthi and Jun, Thanks for the information. I will keep digging. Shan On Wed, Nov 11, 2020 at 6:27 PM SMoorthi-emc <notifications@github.com> wrote:

…

No, I did not. The cause of the problem is in merging with "frac_grid=.true." I think I made changes in sfc_composit routine. The problem is that surface T (and probably q) was becoming too large. Moorthi On Wed, Nov 11, 2020 at 8:10 PM Jun Wang ***@***.***> wrote: > I guess Moorthi might make some changes in get_phi_fv3_run or the gt0 > passed to get_phi_fv3_run. > > On Wed, Nov 11, 2020 at 7:50 PM SMoorthi-emc ***@***.***> > wrote: > > > Shan, > > My branch does things slightly different from what you are doing. I am > > using slmsk=1 if landfrac > 0.0. > > So, using my branch might cause some other issues for you. > > Moorthi > > > > > > On Wed, Nov 11, 2020 at 7:04 PM Shrinivas Moorthi - NOAA Federal < > > ***@***.***> wrote: > > > > > I think my branch does have the fix; I don't remember precisely what > > line. > > > You can try my branch to see if it works for you. > > > I have been running with GFDL MP and it works fine. > > > The only problem I have is that restart is not reproducing - still > trying > > > to chase but CCPP is not that easy (everything works fine under IPD). > > > Moorthi > > > > > > On Wed, Nov 11, 2020 at 4:49 PM shansun6 ***@***.***> > > wrote: > > > > > >> Thank you, Moorthi, for your info. Should I try your ccpp-physics > branch > > >> SM/SM_Oct102020, or is there one routine that I can cherry pick? I > just > > >> want to make the "frac+gfdl" run to complete first. Please advice. > > Thanks! > > >> > > >> — > > >> You are receiving this because you commented. > > >> Reply to this email directly, view it on GitHub > > >> < > > > #268 (comment) > > >, > > >> or unsubscribe > > >> < > > > https://github.com/notifications/unsubscribe-auth/ALLVRYXOBHCI6Q4XADNMHVDSPMBFTANCNFSM4TQH56ZA > > > > > >> . > > >> > > > > > > > > > -- > > > Dr. Shrinivas Moorthi > > > Research Meteorologist > > > Modeling and Data Assimilation Branch > > > Environmental Modeling Center / National Centers for Environmental > > > Prediction > > > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > > > Tel: (301)683-3718 > > > > > > e-mail: ***@***.*** > > > Phone: (301) 683-3718 Fax: (301) 683-3718 > > > > > > > > > -- > > Dr. Shrinivas Moorthi > > Research Meteorologist > > Modeling and Data Assimilation Branch > > Environmental Modeling Center / National Centers for Environmental > > Prediction > > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > > Tel: (301)683-3718 > > > > e-mail: ***@***.*** > > Phone: (301) 683-3718 Fax: (301) 683-3718 > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #268 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AI7D6TIBV3QNU73VZK6ZWIDSPMWOBANCNFSM4TQH56ZA > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALLVRYUQGU4OWQK2JDNN53LSPMYWPANCNFSM4TQH56ZA > > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALORMVTVFJF7LKEVQKZKQ3DSPM2X7ANCNFSM4TQH56ZA> .

ShanSunNOAA · 2020-11-12T06:47:24Z

It appears that the coupled model benchmark case can run successfully with the combination of "frac_grid=T and gfdl MP", with a minimum change of GFS_surface_composites.F90 & GFS_surface_composites.meta from Moorthi's ccpp-physics branch of SM_Oct102020!

Moorthi, thank you so much for your help! If you don't have plan to create a PR just for these two routines, may I do one for you, and what comments do you want to go with this PR? Thanks again.

SMoorthi-emc · 2020-11-12T12:36:49Z

Shan, My draft PR is already there, but I am trying to figure out the restart reproducibility with frac_grid=.true. in the standalone FV3. Can you reproduce? Moorthi

…

On Thu, Nov 12, 2020 at 1:47 AM shansun6 ***@***.***> wrote: It appears that the coupled model benchmark case can run successfully with the combination of "frac_grid=T and gfdl MP", with a minimum change of GFS_surface_composites.F90 & GFS_surface_composites.meta from Moorthi's ccpp-physics branch of SM_Oct102020! Moorthi, thank you so much for your help! If you don't have plan to create a PR just for these two routines, may I do one for you, and what comments do you want to go with this PR? Thanks again. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYWFVV27XVXSZTRN6ODSPOAIVANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

ShanSunNOAA · 2020-11-12T15:36:01Z

I only ran 1 benchmark case with frac_grid=T, which finally completed 24hrs. I will try restart next. Thanks. -Shan On Thu, Nov 12, 2020 at 5:37 AM SMoorthi-emc <notifications@github.com> wrote:

…

Shan, My draft PR is already there, but I am trying to figure out the restart reproducibility with frac_grid=.true. in the standalone FV3. Can you reproduce? Moorthi On Thu, Nov 12, 2020 at 1:47 AM shansun6 ***@***.***> wrote: > It appears that the coupled model benchmark case can run successfully with > the combination of "frac_grid=T and gfdl MP", with a minimum change of > GFS_surface_composites.F90 & GFS_surface_composites.meta from Moorthi's > ccpp-physics branch of SM_Oct102020! > > Moorthi, thank you so much for your help! If you don't have plan to create > a PR just for these two routines, may I do one for you, and what comments > do you want to go with this PR? Thanks again. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALLVRYWFVV27XVXSZTRN6ODSPOAIVANCNFSM4TQH56ZA > > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALORMVXV4G5VFPX4GWWYKATSPPJHBANCNFSM4TQH56ZA> .

ShanSunNOAA · 2020-11-12T21:45:07Z

I ran GFS_surface_composites.F90/meta & sfc_sice.f/meta from SMoorthi-emc/ccpp-physics with the latest develop of ufs-weather-model in the coupled model set up by Denise, and the restart failed at a few lake points on tile3 and 1 lake point on tile2. Most of these lake points are covered with ice to begin with, and no ice left by the time of restart (hr12). I use slmks=floor(landfrac) consistently in the ICs and in FV3, thus these lake points have slmsk of either 0 or 2. However, it still cannot restart reproducibly. Will keep looking. Thanks.

SMoorthi-emc · 2020-11-12T21:59:01Z

Shan, There is another bug that I fixed some time ago that eliminates this error. Since people cherry picked from my code that fix did not make it to develop. Moorthi

…

Sent from my iPhone

On Nov 12, 2020, at 4:45 PM, shansun6 ***@***.***> wrote: I ran GFS_surface_composites.F90/meta & sfc_sice.f/meta from SMoorthi-emc/ccpp-physics with the latest develop of ufs-weather-model in the coupled model set up by Denise, and the restart failed at a few lake points on tile3 and 1 lake point on tile2. Most of these lake points are covered with ice to begin with, and no ice left by the time of restart (hr12). I use slmks=floor(landfrac) consistently in the ICs and in FV3, thus these lake points have slmsk of either 0 or 2. However, it still cannot restart reproducibly. Will keep looking. Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

ShanSunNOAA · 2020-11-13T05:48:43Z

Moorthi, thanks for your info. Let me see if I understand you correctly.

(1) I checked out your branch SM_Oct102020 of ccpp-physics. It has 6 files modified since Dom's commit of f3e6761 on Oct. 9:

physics/GFS_surface_composites.F90
physics/GFS_surface_composites.meta
physics/micro_mg3_0.F90
physics/sfc_sice.f
physics/sfc_sice.meta
physics/GFS_surface_generic.F90

(2) I checked out your branch SM_Oct102020 of FV3. Changes in FV3GFS_io.F90 & GFS_typedefs.F90 seem unrelated to restart, and the rest are IPD related, since Dom's commit on Oct. 9. So I skipped this.

(3) I used these 6 files from (1) above to run with the develop of ufs-weather-model in the coupled mode, it won't reproduce after restart, and the difference remains to be on the icy lake points.

Any suggestions? Thanks,
Shan

SMoorthi-emc · 2020-11-13T12:57:26Z

Shan, While some of my changes in FV3 side are IPD related as I moved some updates inside "#ifdef CCPP" to outside so that the code underneath are not limited to CCPP. By doing so, IPD has the same code as CCPP for the standard global physics. Having said that, you may have noticed that in the coupled model (fractional grid or not) the ice fraction over lakes is lost even if the initial condition has it. The reason is that there is a bug in atmos_model.F90, that I have fixed. This is not just a fractional grid issue or restart issue. I still have the restart issue with fractional grid and CCPP (I do not have this issue with IPD - fractional grid restart reproduces in IPD). Moorthi

…

On Fri, Nov 13, 2020 at 12:49 AM shansun6 ***@***.***> wrote: Moorthi, thanks for your info. Let me see if I understand you correctly. (1) I checked out your branch SM_Oct102020 of ccpp-physics. It has 6 files modified since Dom's commit of f3e6761 on Oct. 9: physics/GFS_surface_composites.F90 physics/GFS_surface_composites.meta physics/micro_mg3_0.F90 physics/sfc_sice.f physics/sfc_sice.meta physics/GFS_surface_generic.F90 (2) I checked out your branch SM_Oct102020 of FV3. Changes in FV3GFS_io.F90 & GFS_typedefs.F90 seem unrelated to restart, and the rest are IPD related, since Dom's commit on Oct. 9. So I skipped this. (3) I used these 6 files from (1) above to run with the develop of ufs-weather-model in the coupled mode, it won't reproduce after restart, and the difference remains to be on the icy lake points. Any suggestions? Thanks, Shan — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYVPKU4E3LAW7RC54H3SPTCERANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

ShanSunNOAA · 2020-11-13T15:56:06Z

Thank Moorthi, for fixing the bug of setting lake ice to zero in atmos_model.F90.
I still suspect this restart issue with fractional grid and CCPP has something to do with slmsk, as most of these failed points (over icy lake points) now failed in the nonfrac case before slmsk was updated as showed by Denise, except now we have 1 failed lake point on tile 2 which has no ice to begin with. But the chase after slmsk went nowhere. Need to find some new clues. Thanks,
Shan

SMoorthi-emc · 2020-11-13T16:16:02Z

Shan, As I wrote, I still have issues with restart with frac_grid=T and CCPP. I don't have this issue with IPD. I am also debugging still - have not given up yet. I also am aware of other inconsistencies related to fractional grid - but that is for the future. Moorthi

…

On Fri, Nov 13, 2020 at 10:56 AM shansun6 ***@***.***> wrote: Thank Moorthi, for fixing the bug of setting lake ice to zero in atmos_model.F90. I still suspect this restart issue with fractional grid and CCPP has something to do with slmsk, as most of these failed points (over icy lake points) now failed in the nonfrac case before slmsk was updated as showed by Denise, except now we have 1 failed lake point on tile 2 which has no ice to begin with. But the chase after slmsk went nowhere. Need to find some new clues. Thanks, Shan — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYXQ44GTYMANGNKFQNLSPVJKLANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

ShanSunNOAA · 2020-11-14T06:19:29Z

Hi Moorthi, I found one problem for the restart with frac_grid=T: tsfco on those failed lake points is very low, much lower than freezing temperature. Upon restart, line 1350 of FV3GFS_io.F90 would reset its value to be "con_tice", which breaks the reproducibility. 1346 if(Model%frac_grid) then ! 3-way composite 1347 !$omp parallel do default(shared) private(nb, ix, tem, tem1) 1348 do nb = 1, Atm_block%nblks 1349 do ix = 1, Atm_block%blksz(nb) 1350 Sfcprop(nb)%tsfco(ix) = max(con_tice, Sfcprop(nb)%tsfco(ix)) I suggest we comment out this line highlighted yellow above for now. Once I did that, I was able to reproduce after restart. Let me know what you think. Thanks, Shan On Fri, Nov 13, 2020 at 9:16 AM SMoorthi-emc <notifications@github.com> wrote:

…

Shan, As I wrote, I still have issues with restart with frac_grid=T and CCPP. I don't have this issue with IPD. I am also debugging still - have not given up yet. I also am aware of other inconsistencies related to fractional grid - but that is for the future. Moorthi On Fri, Nov 13, 2020 at 10:56 AM shansun6 ***@***.***> wrote: > Thank Moorthi, for fixing the bug of setting lake ice to zero in > atmos_model.F90. > I still suspect this restart issue with fractional grid and CCPP has > something to do with slmsk, as most of these failed points (over icy lake > points) now failed in the nonfrac case before slmsk was updated as showed > by Denise, except now we have 1 failed lake point on tile 2 which has no > ice to begin with. But the chase after slmsk went nowhere. Need to find > some new clues. Thanks, > Shan > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALLVRYXQ44GTYMANGNKFQNLSPVJKLANCNFSM4TQH56ZA > > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALORMVS2ZH2GSLB6KDGLKX3SPVLVFANCNFSM4TQH56ZA> .

SMoorthi-emc · 2020-11-14T11:50:24Z

Shan, I guess I have other changes. Yesterday I found that I can reproduce in REPRO=Y mode. Moorrhi

…

Sent from my iPhone

On Nov 14, 2020, at 1:19 AM, shansun6 ***@***.***> wrote: Hi Moorthi, I found one problem for the restart with frac_grid=T: tsfco on those failed lake points is very low, much lower than freezing temperature. Upon restart, line 1350 of FV3GFS_io.F90 would reset its value to be "con_tice", which breaks the reproducibility. 1346 if(Model%frac_grid) then ! 3-way composite 1347 !$omp parallel do default(shared) private(nb, ix, tem, tem1) 1348 do nb = 1, Atm_block%nblks 1349 do ix = 1, Atm_block%blksz(nb) 1350 Sfcprop(nb)%tsfco(ix) = max(con_tice, Sfcprop(nb)%tsfco(ix)) I suggest we comment out this line highlighted yellow above for now. Once I did that, I was able to reproduce after restart. Let me know what you think. Thanks, Shan On Fri, Nov 13, 2020 at 9:16 AM SMoorthi-emc ***@***.***> wrote: > Shan, > As I wrote, I still have issues with restart with frac_grid=T and CCPP. > I don't have this issue with IPD. > I am also debugging still - have not given up yet. > I also am aware of other inconsistencies related to fractional grid - but > that is for the future. > Moorthi > > On Fri, Nov 13, 2020 at 10:56 AM shansun6 ***@***.***> > wrote: > > > Thank Moorthi, for fixing the bug of setting lake ice to zero in > > atmos_model.F90. > > I still suspect this restart issue with fractional grid and CCPP has > > something to do with slmsk, as most of these failed points (over icy lake > > points) now failed in the nonfrac case before slmsk was updated as showed > > by Denise, except now we have 1 failed lake point on tile 2 which has no > > ice to begin with. But the chase after slmsk went nowhere. Need to find > > some new clues. Thanks, > > Shan > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #268 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/ALLVRYXQ44GTYMANGNKFQNLSPVJKLANCNFSM4TQH56ZA > > > > . > > > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > > — > You are receiving this because you were assigned. > Reply to this email directly, view it on GitHub > <#268 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALORMVS2ZH2GSLB6KDGLKX3SPVLVFANCNFSM4TQH56ZA> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

DeniseWorthen · 2020-11-14T12:04:41Z

Shouldn't we try to figure out why the tsfco temperatures are below freezing?

ShanSunNOAA · 2020-11-14T15:39:04Z

Denise, good point. These below-freezing water temperature occurred over lake points that started without ice. Without a lake model, no new lake ice can form at lake points with 100% open water, since sfc_sice.f will skip points without ice. Maybe gcycle can introduce ice at these cold lake points. How about setting water temperature not-below-freezing only at the initial time, and not at restart, to guaranteer restart reproducibility?

junwang-noaa · 2020-11-15T01:55:27Z

Shan, thanks for finding this issue! I think it is good to check 1) the initial value of tsfco on this lake point and confirm it is decreasing to below freezing during forecast time before restart. 2) later gcycle changes the open water temp and has ice information put on this point, otherwise we may still have issues on below freezing open water points.

…

On Sat, Nov 14, 2020 at 10:39 AM shansun6 ***@***.***> wrote: Denise, good point. These below-freezing water temperature occurred over lake points that started without ice. Without a lake model, no new lake ice can form at lake points with 100% open water, since sfc_sice.f will skip points without ice. Maybe gcycle can introduce ice at these cold lake points. How about setting water temperature not-below-freezing only at the initial time, and not at restart, to guaranteer restart reproducibility? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TOIUP3Q7ZQES4UAHP3SP2QCJANCNFSM4TQH56ZA> .

SMoorthi-emc · 2020-11-15T02:24:23Z

I am adding a potential fix in gcycle.F90 as " if (slifcs(len) > 1.9_kind_phys) then Sfcprop(nb)%tsfco(ix) = con_tice endif" Moorthi

…

On Sat, Nov 14, 2020 at 8:55 PM Jun Wang ***@***.***> wrote: Shan, thanks for finding this issue! I think it is good to check 1) the initial value of tsfco on this lake point and confirm it is decreasing to below freezing during forecast time before restart. 2) later gcycle changes the open water temp and has ice information put on this point, otherwise we may still have issues on below freezing open water points. On Sat, Nov 14, 2020 at 10:39 AM shansun6 ***@***.***> wrote: > Denise, good point. These below-freezing water temperature occurred over > lake points that started without ice. Without a lake model, no new lake ice > can form at lake points with 100% open water, since sfc_sice.f will skip > points without ice. Maybe gcycle can introduce ice at these cold lake > points. How about setting water temperature not-below-freezing only at the > initial time, and not at restart, to guaranteer restart reproducibility? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AI7D6TOIUP3Q7ZQES4UAHP3SP2QCJANCNFSM4TQH56ZA > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYWCDJDNQ72OPQRBU23SP4YJVANCNFSM4TQH56ZA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

ShanSunNOAA · 2020-11-15T04:55:29Z

Moorthi, good that you added this potential fix. However, it won't help these three lake points on tile3 that failed restart reproducibility, since no points have ice to begin with or during the 24hrs. [image: Screen Shot 2020-11-14 at 8.49.25 PM.png] I have added a "if" to this statement in FV3GFS_io.F90, so that guard check will only occur during the initial time: if( Model%phour < 1.e-7) Sfcprop(nb)%tsfco(ix) = max(con_tice, Sfcprop(nb)%tsfco(ix)) If the model generates water temp far below freezing, it won't be reset to con_tice during restart, in order to be able to reproduce after restart. Make sense? Thanks, Shan On Sat, Nov 14, 2020 at 7:24 PM SMoorthi-emc <notifications@github.com> wrote:

…

I am adding a potential fix in gcycle.F90 as " if (slifcs(len) > 1.9_kind_phys) then Sfcprop(nb)%tsfco(ix) = con_tice endif" Moorthi On Sat, Nov 14, 2020 at 8:55 PM Jun Wang ***@***.***> wrote: > Shan, thanks for finding this issue! I think it is good to check 1) the > initial value of tsfco on this lake point and confirm it is decreasing to > below freezing during forecast time before restart. 2) later gcycle changes > the open water temp and has ice information put on this point, otherwise we > may still have issues on below freezing open water points. > > On Sat, Nov 14, 2020 at 10:39 AM shansun6 ***@***.***> > wrote: > > > Denise, good point. These below-freezing water temperature occurred over > > lake points that started without ice. Without a lake model, no new lake > ice > > can form at lake points with 100% open water, since sfc_sice.f will skip > > points without ice. Maybe gcycle can introduce ice at these cold lake > > points. How about setting water temperature not-below-freezing only at > the > > initial time, and not at restart, to guaranteer restart reproducibility? > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #268 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AI7D6TOIUP3Q7ZQES4UAHP3SP2QCJANCNFSM4TQH56ZA > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #268 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALLVRYWCDJDNQ72OPQRBU23SP4YJVANCNFSM4TQH56ZA > > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALORMVXK4RORR3ENWASMMLDSP43WHANCNFSM4TQH56ZA> .

SMoorthi-emc · 2020-11-15T12:24:58Z

Well, part of the problem here is that sfccycle is yet to be fixed for the fractional grid. Moorthi

…

Sent from my iPhone

On Nov 14, 2020, at 11:55 PM, shansun6 ***@***.***> wrote: Moorthi, good that you added this potential fix. However, it won't help these three lake points on tile3 that failed restart reproducibility, since no points have ice to begin with or during the 24hrs. [image: Screen Shot 2020-11-14 at 8.49.25 PM.png] I have added a "if" to this statement in FV3GFS_io.F90, so that guard check will only occur during the initial time: if( Model%phour < 1.e-7) Sfcprop(nb)%tsfco(ix) = max(con_tice, Sfcprop(nb)%tsfco(ix)) If the model generates water temp far below freezing, it won't be reset to con_tice during restart, in order to be able to reproduce after restart. Make sense? Thanks, Shan On Sat, Nov 14, 2020 at 7:24 PM SMoorthi-emc ***@***.***> wrote: > I am adding a potential fix in gcycle.F90 as > " if (slifcs(len) > 1.9_kind_phys) then > Sfcprop(nb)%tsfco(ix) = con_tice > endif" > Moorthi > > On Sat, Nov 14, 2020 at 8:55 PM Jun Wang ***@***.***> wrote: > > > Shan, thanks for finding this issue! I think it is good to check 1) the > > initial value of tsfco on this lake point and confirm it is decreasing to > > below freezing during forecast time before restart. 2) later gcycle > changes > > the open water temp and has ice information put on this point, otherwise > we > > may still have issues on below freezing open water points. > > > > On Sat, Nov 14, 2020 at 10:39 AM shansun6 ***@***.***> > > wrote: > > > > > Denise, good point. These below-freezing water temperature occurred > over > > > lake points that started without ice. Without a lake model, no new lake > > ice > > > can form at lake points with 100% open water, since sfc_sice.f will > skip > > > points without ice. Maybe gcycle can introduce ice at these cold lake > > > points. How about setting water temperature not-below-freezing only at > > the > > > initial time, and not at restart, to guaranteer restart > reproducibility? > > > > > > — > > > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > > > < > > > #268 (comment) > > >, > > > or unsubscribe > > > < > > > https://github.com/notifications/unsubscribe-auth/AI7D6TOIUP3Q7ZQES4UAHP3SP2QCJANCNFSM4TQH56ZA > > > > > > . > > > > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #268 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/ALLVRYWCDJDNQ72OPQRBU23SP4YJVANCNFSM4TQH56ZA > > > > . > > > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > > — > You are receiving this because you were assigned. > Reply to this email directly, view it on GitHub > <#268 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALORMVXK4RORR3ENWASMMLDSP43WHANCNFSM4TQH56ZA> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

…was fixed by Moorthi's modifications in ccpp/physics; -- Restart in the coupled mode with the default physics is reproducible, when (1) bad water temperature is only filtered at the initial time; (2) stop initializing fice to zero everywhere in the FV3 cap in order to keep lake ice untouched; These address Issues ufs-community#268, ufs-community#285 & ufs-community#286. Co-authored-by: Shrinivas Moorthi <shrinivas.moorthi@noaa.gov> Co-authored-by: Denise Worthen <Denise.Worthen@noaa.gov>

ShanSunNOAA · 2020-12-02T22:59:03Z

This issue is resolved in the commit today. Thanks Moorthi for fixing this bug.

… zorl interstitial, ocn -> wat, merra2 threading (ufs-community#279) * changed .gitmodules to point to merra2 ccpp/physics * remove GFDL_atmos_cubed_sphere and ccpp-framework from .git module * remove IPD gfsphysics * Update .gitmodules and submodule pointer for ccpp-physics for code review and testing * Remove interstitial zorl composites * Update .gitmodules and submodule pointer fpor ccpp-physics for code review and testing * Remove or replace references to IPD in comments in atmos_model.F90 * Initialize Sfcprop%zorlx to clear_val instead of huge * Update submodule pointer for ccpp-physics * Rename Fortran variables and CCPP standard names / long names of surface composites from ocean to water * Rename Sfcprop%zorlw to Sfcprop%zorlwav * Rename Sfcprop%zorlo to Sfcprop%zorlw * update submodule pointer for ccpp-physics * Revert change to .gitmodules and update submodule pointer for ccpp-physics Co-authored-by: anning.cheng <anning.cheng@noaa.gov>

* Remove all references to /lfs3 on Jet * Add Ben and Ratko to the CODEOWNERS file * Replace hard-coded make_orog module file with build-level module file in UFS_UTILS * Remove hard-coded make_sfc_climo module file * Rename all FV3-SAR and SAR-FV3 to FV3-LAM, rename all JPgrid to ESGgrid. Remove fix files in regional_workflow and source from fix_am and EMC_post. * Add alpha/kappa parameter back in exregional_make_grid.sh * Remove dash from FV3LAM_wflow.xml * Change FIXam to FIXgsm to source Thompson CCN file * Remove old, unused grid stanza from exregional_run_post.sh * Change Jet locations of fix_am/fix_orog to EMC paths

Enables the build and test workflow to run on the Hera and Jet platforms.

ShanSunNOAA added the bug Something isn't working label Nov 10, 2020

junwang-noaa assigned ShanSunNOAA Nov 11, 2020

ShanSunNOAA closed this as completed Dec 2, 2020

epic-cicd-jenkins pushed a commit that referenced this issue Apr 17, 2023

Add Hera and Jet to pipeline (#268)

bb1ee17

Enables the build and test workflow to run on the Hera and Jet platforms.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coupled run using benchmark at C384 crashed when frac_grid=T #268

Coupled run using benchmark at C384 crashed when frac_grid=T #268

ShanSunNOAA commented Nov 10, 2020

yangfanglin commented Nov 10, 2020

ShanSunNOAA commented Nov 10, 2020 via email

ShanSunNOAA commented Nov 11, 2020

junwang-noaa commented Nov 11, 2020

ShanSunNOAA commented Nov 11, 2020

SMoorthi-emc commented Nov 11, 2020 via email

ShanSunNOAA commented Nov 11, 2020

SMoorthi-emc commented Nov 12, 2020 via email

SMoorthi-emc commented Nov 12, 2020 via email

junwang-noaa commented Nov 12, 2020 via email

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 13, 2020

SMoorthi-emc commented Nov 13, 2020 via email

ShanSunNOAA commented Nov 13, 2020

SMoorthi-emc commented Nov 13, 2020 via email

ShanSunNOAA commented Nov 14, 2020 via email

SMoorthi-emc commented Nov 14, 2020 via email

DeniseWorthen commented Nov 14, 2020

ShanSunNOAA commented Nov 14, 2020

junwang-noaa commented Nov 15, 2020 via email

SMoorthi-emc commented Nov 15, 2020 via email

ShanSunNOAA commented Nov 15, 2020 via email

SMoorthi-emc commented Nov 15, 2020 via email

ShanSunNOAA commented Dec 2, 2020

Coupled run using benchmark at C384 crashed when frac_grid=T #268

Coupled run using benchmark at C384 crashed when frac_grid=T #268

Comments

ShanSunNOAA commented Nov 10, 2020

Description

To Reproduce:

yangfanglin commented Nov 10, 2020

ShanSunNOAA commented Nov 10, 2020 via email

ShanSunNOAA commented Nov 11, 2020

junwang-noaa commented Nov 11, 2020

ShanSunNOAA commented Nov 11, 2020

SMoorthi-emc commented Nov 11, 2020 via email

ShanSunNOAA commented Nov 11, 2020

SMoorthi-emc commented Nov 12, 2020 via email

SMoorthi-emc commented Nov 12, 2020 via email

junwang-noaa commented Nov 12, 2020 via email

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 12, 2020

SMoorthi-emc commented Nov 12, 2020 via email

ShanSunNOAA commented Nov 13, 2020

SMoorthi-emc commented Nov 13, 2020 via email

ShanSunNOAA commented Nov 13, 2020

SMoorthi-emc commented Nov 13, 2020 via email

ShanSunNOAA commented Nov 14, 2020 via email

SMoorthi-emc commented Nov 14, 2020 via email

DeniseWorthen commented Nov 14, 2020

ShanSunNOAA commented Nov 14, 2020

junwang-noaa commented Nov 15, 2020 via email

SMoorthi-emc commented Nov 15, 2020 via email

ShanSunNOAA commented Nov 15, 2020 via email

SMoorthi-emc commented Nov 15, 2020 via email

ShanSunNOAA commented Dec 2, 2020