all zero values in atmImp_Faxa_lwnet over ocean for cpld_control_p7 test

[uturuncoglu]

Description

atmImp_Faxa_lwnet variable in the mediator history output has all zero over the ocean. This is probably passed to the other model components such as MOM6 that could cause an issue.

To Reproduce:

What compilers/machines are you seeing this with? Intel+MPT on Cheyenne
Give explicit steps to reproduce the behavior.

1. checkout ufs-wearher model (head of develop)
2. add mediator history phase to the run sequence in nems.configure (MED med_phases_history_write)
3. run model
4. check atmImp_Faxa_lwnet in the mediator history file.

Output

[DeniseWorthen]

This can also be seen in the output files for MOM6, without using the mediator history files. In the file ocn_2021_03_23_03.nc, which is the last MOM6 output file for the cpld_control_p7 test, the field lw shows constant values of 0.0 except for 6 non-zero points on the coast of Baffin Is.

[uturuncoglu]

@DeniseWorthen it is weird that this is not caught with the RTs. I was expecting to see difference in the ocean fields and RT needs to be failed. Maybe we need to think about it.

[DeniseWorthen]

I confirmed that the issue arose at commit b87cdaa. That commit changed baselines. @SMoorthi-emc could you please take a look?

[uturuncoglu]

@DeniseWorthen That explains why this is not caught by the RTs but again I think that there is a gap in the current RT system that needs to be considered for the future improvement of the testing strategy. Maybe a set of graphical output could be generated along with the RTs. As I know this is done with the CESM but probably Mariana has better idea. At least for the single platform, a diagnostic tool can be used to create set of plots of difference from the previous baseline, publish in a web site that all the developers/users could look at visually to see any possible issue. It would be also helpful for the users to see the actual baseline changes through the output.

[DeniseWorthen]

@uturuncoglu I agree. A system like that would be very useful.

[SMoorthi-emc]

I confirm that this is true in my runs too😞Moorthi

…

On Tue, Nov 16, 2021 at 12:12 PM Denise Worthen ***@***.***> wrote: I confirmed that the issue arose at commit b87cdaa <b87cdaa>. That commit changed baselines. @SMoorthi-emc <https://github.com/SMoorthi-emc> could you please take a look? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYWHN4FKISLN2AOG7MTUMKGFHANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[uturuncoglu]

@DeniseWorthen I am not sure but it would be nice to create another issue for the discussion about RT regression system. We could leverage form NCAR's experience. Of course they are looking for different properties such as long term drift etc. but it would be still helpful. Maybe we could have a meeting just for focusing this. I think there are bunch of existing data analysis script in NOAA side that could be used for this purpose. It is just a mater of collecting them in a separate GitHub repository and integrate with the existing RT system.

[SMoorthi-emc]

I looked at the ATm sfc files and the fluxes are fine there. On Tue, Nov 16, 2021 at 12:20 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote:

…

I confirm that this is true in my runs too😞Moorthi On Tue, Nov 16, 2021 at 12:12 PM Denise Worthen ***@***.***> wrote: > I confirmed that the issue arose at commit b87cdaa > <b87cdaa>. > That commit changed baselines. @SMoorthi-emc > <https://github.com/SMoorthi-emc> could you please take a look? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#914 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALLVRYWHN4FKISLN2AOG7MTUMKGFHANCNFSM5IEYQSIQ> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[junwang-noaa]

@uturuncoglu That is a good idea! Looking forward to seeing what metrics are used for the coupled system and how the graphics are integrated in the test systems in CESM.

[DeniseWorthen]

I ran commits 5a20ea4 (prior to yours) and then b87cdaa (yours). I looked at the coupler history files for the two, and saw the impact only after the emissivity commit.

[uturuncoglu]

@junwang-noaa maybe we could use one of the Friday's call for this purpose. It would be very useful.

[junwang-noaa]

Great, let's include Rocky to see if he can set up a meeting. @rsdunlapiv

[SMoorthi-emc]

It appears that there is some logic error in the atmos_model.F90 where fields are collected for export to the ocean. I put print just after collecting dlwsfci and it did not find that print. I do not completely follow the logic in this routine. I also put print on the ccpp side for the same array and the values are fine there. Moorthi On Tue, Nov 16, 2021 at 12:30 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote:

…

I looked at the ATm sfc files and the fluxes are fine there. On Tue, Nov 16, 2021 at 12:20 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote: > I confirm that this is true in my runs too😞Moorthi > > > On Tue, Nov 16, 2021 at 12:12 PM Denise Worthen ***@***.***> > wrote: > >> I confirmed that the issue arose at commit b87cdaa >> <b87cdaa>. >> That commit changed baselines. @SMoorthi-emc >> <https://github.com/SMoorthi-emc> could you please take a look? >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#914 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ALLVRYWHN4FKISLN2AOG7MTUMKGFHANCNFSM5IEYQSIQ> >> . >> Triage notifications on the go with GitHub Mobile for iOS >> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> >> or Android >> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. >> >> > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[DeniseWorthen]

You can set dbug_flag = 6 in nems.configure; doing so will turn on the state_diagnose function for FV3.

That confirms that in the export state, the values are zero:

20211116 112729.170 INFO             PET100 (FV3: state_diagnose) :ES:mean_net_lw_flx    0.000000      0.000000      0.000000

compared to previously

20211116 112055.870 INFO             PET100 (FV3: state_diagnose) :ES:mean_net_lw_flx   -90.20896     -28.57923     -21555.13

[SMoorthi-emc]

Well, that does not help in finding the bug. I did not make any change to this routine in my commit. Unfortunately, this routine uses ESMF variables which can not be printed to figure out what is going on. This routine was updated by someone else recently. The array "dlwsfci_cpl" contains valid data, but is not used, it appears.

…

On Tue, Nov 16, 2021 at 1:34 PM Denise Worthen ***@***.***> wrote: You can set dbug_flag = 6 in nems.configure; doing so will turn on the state_diagnose function for FV3. That confirms that in the export state, the values are zero: 20211116 112729.170 INFO PET100 (FV3: state_diagnose) :ES:mean_net_lw_flx 0.000000 0.000000 0.000000 compared to previously 20211116 112055.870 INFO PET100 (FV3: state_diagnose) :ES:mean_net_lw_flx -90.20896 -28.57923 -21555.13 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYXNWBLNOZFYDVWFEPDUMKP3TANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[DeniseWorthen]

Yes, but the two state_diagnose lines I wrote out were for your commit and the one immediately prior.

[SMoorthi-emc]

Maybe, but I did not modify this routine and the data that should have been exported is non-zero; I wrote out min and max values and they are nonzero. Also, I put the same print just after the line copying the date to be exported, but the run never reached that line. So, I guess there is a logic error (not introduced in my commit.

…

On Tue, Nov 16, 2021 at 1:53 PM Denise Worthen ***@***.***> wrote: Yes, but the two state_diagnose lines I wrote out were for your commit and the one immediately prior. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYR3N5AGTCFGNWL2WELUMKSD5ANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[SMoorthi-emc]

I checked for the fields coming inside the loop "do n=1, size(exportFields)" and it appears that there are no "inst_" fields coming in. On Tue, Nov 16, 2021 at 1:58 PM Shrinivas Moorthi - NOAA Federal < ***@***.***> wrote:

…

Maybe, but I did not modify this routine and the data that should have been exported is non-zero; I wrote out min and max values and they are nonzero. Also, I put the same print just after the line copying the date to be exported, but the run never reached that line. So, I guess there is a logic error (not introduced in my commit. On Tue, Nov 16, 2021 at 1:53 PM Denise Worthen ***@***.***> wrote: > Yes, but the two state_diagnose lines I wrote out were for your commit > and the one immediately prior. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#914 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ALLVRYR3N5AGTCFGNWL2WELUMKSD5ANCNFSM5IEYQSIQ> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[uturuncoglu]

I wonder if any update about this issue? It would be hard for me to compare CMEPS calculated fluxes without a fix. Please also let me know if there is a version of the ufs-weather-model that I could use and has no this issue.

[junwang-noaa]

@uturuncoglu You can use revision 5a20ea4, which does not have this issue.

[uturuncoglu]

@junwang-noaa Okay. That is great. Thanks.

[SMoorthi-emc]

I think I found the problem. Testing now to see if it is the problem. If yes, fixing will require one line change.

…

On Thu, Nov 18, 2021 at 12:50 PM Jun Wang ***@***.***> wrote: @uturuncoglu <https://github.com/uturuncoglu> You can use revision 5a20ea4 <5a20ea4>, which does not have this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYUIUETKVZUZXM2RQLDUMU4E7ANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[SMoorthi-emc]

Yes, it works!

[AvichalMehra-NOAA]

Excellent Moorthi! Avichal.

…

On Thu, Nov 18, 2021 at 3:01 PM SMoorthi-emc ***@***.***> wrote: Yes, it works! [image: Net_lw_into_ocn] <https://user-images.githubusercontent.com/47667426/142488065-5eb2908b-95ec-4c41-921b-438a9a97fc1b.PNG> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJ6ZTFB2TB7LTMBI37TCYL3UMVLPZANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Avichal Mehra Chief, Dynamics and Coupled Modeling Group Modeling and Data Assimilation Branch NOAA/NWS/NCEP/EMC Ph. 301-683-3746 Fax: 301-683-3703 ***@***.***

[DeniseWorthen]

@SMoorthi-emc You said earlier in the thread that the ATM sfc flux fields looked fine. But the figure also looks like a sfc flux field? So was this field zero earlier?

[SMoorthi-emc]

I found the bug and fixed it in my branch yesterday and created an issue.
I also attached an updated figure in another thread yesterday, but no one seems to have seen it.
Fix is a "one' line change in GFS_Typedef. The error occurred because of initialization of interstitial variables.

[SMoorthi-emc]

Sorry, this is the thread - I see that Avichal and Denise have seen it. The above figure is from the MOM6 output (I am not sure if it is on a tri-polar grid). I also don't know how to template this output in grads. Note that this is net long wave - initially I was looking at the downward long wave in the atm files, which looked fine.

[DeniseWorthen]

This is the tripole grid if it is MOM6 output. I can also tell from the blank space around 120W, north of 70N. That is actually land in the tripole grid, so it shows up as no values.

Thanks for the explanation of why earlier you said the sfc flux files looked fine.

[SMoorthi-emc]

Comparing with net LW in pgb master file, they look very similar (pgb file values are 3 hour mean on Gaussian grid).

…

On Fri, Nov 19, 2021 at 7:36 AM Denise Worthen ***@***.***> wrote: This is the tripole grid if it is MOM6 output. I can also tell from the blank space around 120W, north of 70N. That is actually land in the tripole grid, so it shows up as no values. Thanks for the explanation of why earlier you said the sfc flux files looked fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYWS66GMPZAZ6KKRKTTUMZAELANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[junwang-noaa]

@SMoorthi-emc Thanks for fixing this issue! Since the fix is in GFS_Typedef, are you going to create an FV3 PR associated with the fv3 issue you created?

[SMoorthi-emc]

Maybe or if someone wants to combine this another PR, it is ok with me as this is only removing one line "Interstitial%semis_water = clear_val" Moorthi

…

On Fri, Nov 19, 2021 at 8:45 AM Jun Wang ***@***.***> wrote: @SMoorthi-emc <https://github.com/SMoorthi-emc> Thanks for fixing this issue! Since the fix is in GFS_Typedef, are you going to create an FV3 PR associated with the fv3 issue you created? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYS5VIH2XM6WYI3NPW3UMZIFPANCNFSM5IEYQSIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[junwang-noaa]

Thanks, we will combine this change with the next PR that needs new baseline.

[DeniseWorthen]

I also checked the analogous file which was shown in the original issue and verified that the field is now non-zero

:

[climbfuji]

@SMoorthi-emc (@DeniseWorthen @junwang-noaa) I want to double check that the values are not only non-zero, but also correct. I noted you said that you removed a singe line

Interstitial%semis_water     = clear_val

Where? In GFS_typedefs.F90 in the interstitial_phys_reset routine? Note that the GFS_interstitial_type is not a persistent variable and the resetting is done on purpose. The variable gets reused for every block that an MPI task owns and does not contain valid data at the end of the physics time step. Everything that is in the GFS_interstitial_type is transient and only valid while within the physics or radiation group of the SDF for the current block number.

If you need to back this data out directly, i.e. without copying it into another array within the physics group, then semis_water needs to be moved from the GFS_interstitial_type to another DDT.

I hope my explanation makes sense ...

[SMoorthi-emc]

Dom, you are right I was removing from interstitial reset, because it was zeroing it out.. This is the result of complex CCPP implementation.
I could have added sfcemis_water to Sfcprop just like I did for ice, but water emissivity is just a constant defined in surface radiation.f and there is no point adding it to the restart fle. There are alternate ways to handle this, but the easiest for now is just to remove this line.

[SMoorthi-emc]

If I forgot to mention, CCPP is not the easiest code to work with.

[DeniseWorthen]

I checked the same field at hash 5a20ea4 (one prior to the emissivity commit) and the above figure is nearly identical. So, presuming that the lwnet was correct before, the fix seems to be correct.

[SMoorthi-emc]

Thanks for checking.

…

Sent from my iPhone

On Nov 19, 2021, at 3:59 PM, Denise Worthen ***@***.***> wrote: I checked the same field at hash 5a20ea4 (one prior to the emissivity commit) and the above figure is nearly identical. So, presuming that the lwnet was correct before, the fix seems to be correct. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[climbfuji]

Thanks for checking.
…

Thank you both, good to know that there is an easy fix for it.

[DeniseWorthen]

@uturuncoglu This will be fixed when we merge PR #921 on Monday morning.