CPT tuning effort

[bstephens82]

@vlarson @JulioTBacmeister @adamrher @zarzycki we can put tuning-related developments here.

This link will take you to my "starting point" default run that I used with Zhun's tuner scripts: https://webext.cgd.ucar.edu/F2000climo/zt0/atm/zt0-obs.0001_1/. This run includes

• prognostic momentum fluxes
• taus code
• Colin's patch
• CLUBB booleans discussed here
• Colin's tunable CLUBB parameters from early 2022 (mostly---I have increased the value of C_invrs_tau_N2_xp2 to 0.5 to brighten SWCF)
• fixes for CLUBB's monotonic flux limiter

These should account for most of what @Katetc is bringing in with the new external, although there may be some other minor changes to CLUBB coming in with the new external.

[vlarson]

Colin provided some preliminary plots of output here:

"x021 runs = cam_dev
x222 runs = cam_dev + predict_upwp + diag_Lscale_from_tau

In x222 runs ("new") runs, that increase in precip near the equator appears to be due to lower ZM% and therefore more precip from PUMAS (see FREQZM and PRECZM_FRACTION). x222 runs also has lower CAPE, but more CLDLOW and total cloud water path.

So for "taus + upwp" (untuned!) in the tropics, cloudier, (slightly) more stable, clouds are more bottom-heavy, PUMAS rains more vs. ZM.

AMWG diags here.
https://colinzarzycki.com/cpt/diag/FHIST-ne30-free-x021/FHIST-ne30-free-x021-obs/
https://colinzarzycki.com/cpt/diag/FHIST-ne30-free-x222/FHIST-ne30-free-x222-obs/"

[vlarson]

"x021 runs = cam_dev x222 runs = cam_dev + predict_upwp + diag_Lscale_from_tau

In x222 runs ("new") runs, that increase in precip near the equator appears to be due to lower ZM% and therefore more precip from PUMAS (see FREQZM and PRECZM_FRACTION). x222 runs also has lower CAPE, but more CLDLOW and total cloud water path.

So for "taus + upwp" (untuned!) in the tropics, cloudier, (slightly) more stable, clouds are more bottom-heavy, PUMAS rains more vs. ZM.

AMWG diags here. colinzarzycki.com/cpt/diag/FHIST-ne30-free-x021/FHIST-ne30-free-x021-obs colinzarzycki.com/cpt/diag/FHIST-ne30-free-x222/FHIST-ne30-free-x222-obs"

Based on these AMWG plots, it looks like with the taus code on, SHFLX and LHFLX are larger. I wonder if C_invrs_tau_sfc or C_invrs_tau_shear ought to be boosted.

[bstephens82]

Vince and I looked at some tuner plots this morning. C_uu_shr has a small value in the default run---we could try a run with a larger value of C_uu_shr (maybe 0.5) and see what that does. We could also try a few additional runs extending the range of C_uu_shr for the tuner.
We also found a decent-looking result with parameter values:

c_uu_shr = 0.225
gamma_coef = 0.573
altitude_threshold = 0.374
invrs_tau_shear = 0.224

so we could try that. This seemed to help a lot with Hawaii, DYCOMS, and VOCALS. These numbers came from tuning C_uu_shr, gamma_coef, altitude_threshold, and c_invrs_tau_shear, with DYCOMS and VOCALS weighted 4 and Hawaii weighted 8 (globals weighted 12).

Since this was a first effort, probably worth looking at the ranges of all the parameters and consider whether they should be extended, etc.
Also might be worth it to do a sanity check compare the atm_in files. We are getting a lot of hyperbolic results (i.e. tuning a parameter in either direction makes the result worse), what could be causing that?

[bstephens82]

Met w/ Vince 1/25/23. Vince says do a run with and without taus and then do the AMWG diagnostics model v model and Zhun's plots too. Model v model plots may contain omega comparison. Need to figure out why it's going deep in eastern Pacific; check CLUBB's wp2 variable in this connection.

What's going on over DYCOMS region in the profile plots? Also compare the red and blue arrows in Zhun's tuner plots (requires processing the new runs with the diagnostics code).

[bstephens82]

I used Zhun's tuner with a set of parameters that was supposed to improve DYCOMS and the eastern Pacific (EP), with Hawaii weighted at a medium value, and here are the results from that run (zt14) compared to obs and compared to my default (zt0) run (link for that one above). The eastern Pacific did improve but DYCOMS and Hawaii got worse, and went in the opposite direction from what the tuner predicted (why?). In other words, things dimmed in the EP as desired, but this also dimmed Sc which is the opposite of what we want.

zt14 minus obs:
https://webext.cgd.ucar.edu/F2000climo/zt14/atm/zt14-obs.0001_1/

zt14 minus zt0:
https://webext.cgd.ucar.edu/F2000climo/zt14/atm/zt14.0001_1-zt0.0001_1/

[adamrher]

This is consistent with my experience hand tuning; adjusting the skewness via clubb_c8 or clubb_gamma impacts the clouds everywhere. I think we consider the dycoms and vocals regions the most important to get right for radiation balance, and so getting the Sc decks right really determines what those parameter values are.

Which I think is partly why these skewness dependent parameters came about, to provide some more selectiveness. Would it be possible to add some of the skewness dependent parameters to the tuner? c11b is one I've used for tuning, and I don't think it's a "tau" parameter (this is for the w'3 eqn).

[bstephens82]

Sure I can try some tuner runs with those parameters.

[bstephens82]

Hi @adamrher I did a tuner test using the skewness-dependent parameters I had access to (I hadn't included C7 initially so that one was not included but I have the necessary files now):
https://webext.cgd.ucar.edu/F2000climo/zt16/atm/zt16-obs.0001_1/

The parameters I tuned for this run are as follows (with their default values after the exclamation mark):

clubb_c8=3.51 !3.440377776099962
clubb_c_invrs_tau_sfc=0.13 !0.12743072568015346
clubb_c_invrs_tau_shear=0.354 !0.12502726304767026
clubb_c_invrs_tau_n2_xp2=0.405 !0.5
clubb_c_invrs_tau_n2_clear_wp3=0.479 !1.0000
clubb_c11=0.29 !0.31057411754034614
clubb_c11b=0.351 !0.3250718127387944

This brightened things significantly, well past where we would ultimately want it to be, but I am curious about your opinion since you said the DYCOMS and VOCALS regions are important to get right and I targeted those regions in this tuner test.

[bstephens82]

I talked with @swrneale the other day about how to possibly get ZM to be less active in the eastern Pacific (EP)---a look at some profile plots does suggest that omega is too strong in this region and things are too cloudy, but CLUBB is mostly active near the surface so presumably this is mainly related to ZM---and his first suggestion was to set the flag zmconv_parcel_pbl = .false., which is normally true by default for runs using cam_development. Here is the output of that run (this link includes both comparison to obs and comparison to my default zt0 run).

https://webext.cgd.ucar.edu/F2000climo/zt11/atm/

This flag does not seem to have had much impact in the deep tropical EP, but did brighten things up by a few W/m2 in the global mean. @swrneale do you have other suggestions for things to modify within ZM that might help to dim the EP?

[swrneale]

@bstephens82 this is a challenge as the entrainment rate and stability are cranked up to probably their highest level. Do you have the climo files for the existing cam-dev 'control' and your taus code tuning effort, just so I can see how ZM contributions have changed?

[bstephens82]

@swrneale here are a couple of links that might be helpful. This one is the out-of-the-box cam_dev L58 run from the same tag that I used to produce my zt11 run from above:
https://webext.cgd.ucar.edu/F2000climo/dflt078/atm/dflt078-obs.0001_1/

And here is a link to plots showing the difference between zt11 and dflt078:
https://webext.cgd.ucar.edu/F2000climo/zt11/atm/zt11.0001_1-dflt078.0001_1/

Hopefully this helps you draw conclusions about how ZM contributions have changed?

[vlarson]

@bstephens82, is it worth creating a table that documents the differences in configuration (flag settings, parameter values, etc.) between the runs that you're doing? And that adds links to plots? I don't know what configuration change caused the eastern Pacific to become so bright in the first place.

[bstephens82]

@vlarson I have been keeping a spreadsheet of all my tests which I've shared with Julio...I sent you the link too. It would be nice to have a linked table here in the thread, when I have some time I will try to create something like that.

[adamrher]

John Truesdale uses a google doc template for tracking simulations. It's not perfect, but it does the job:
https://docs.google.com/document/d/1H0NFB5XXQXP8CehDgTdrM_EfYRhbW_NgyL44KAnSnDk/edit?usp=sharing

[bstephens82]

That's nice, thanks for sharing. I'll try to do something like that when I get to it.

[vlarson]

Ben, it might be worth setting l_smooth_min_max=T and l_smooth_Heaviside_tau_wpxp = T. I doubt that those flag choices will improve the Sc or ITCZ regions, but they might smooth the solutions and parameter dependence, and hence they might make the model easier to tune.

[bstephens82]

I thought I would share one of my latest runs, which I call zt18. For this run I modified Andrew's microphysics parameters and CLUBB's gamma_coef:

clubb_gamma_coef=0.45 !0.5492223674353673
micro_mg_autocon_fact=0.02 !0.005
micro_mg_dcs=0.00025 !0.001

This has strongly brightened the Sc decks (too much in the case of the South American deck), but interestingly has not greatly brightened the eastern Pacific ITCZ relative to my control (it may even look very slightly better than the control). It also has mean SWCF -46.43, which is closer to what I believe is the new CERES-EBAF number, something like -45.8 or so.

Diagnostics here:
https://webext.cgd.ucar.edu/F2000climo/zt18/atm/zt18-obs.0001_1/

[vlarson]

I thought I would share one of my latest runs, which I call zt18. For this run I modified Andrew's microphysics parameters and CLUBB's gamma_coef:

clubb_gamma_coef=0.45 !0.5492223674353673
micro_mg_autocon_fact=0.02 !0.005
micro_mg_dcs=0.00025 !0.001

This has strongly brightened the Sc decks (too much in the case of the South American deck), but interestingly has not greatly brightened the eastern Pacific ITCZ relative to my control (it may even look very slightly better than the control). It also has mean SWCF -46.43, which is closer to what I believe is the new CERES-EBAF number, something like -45.8 or so.

Diagnostics here: webext.cgd.ucar.edu/F2000climo/zt18/atm/zt18-obs.0001_1

Do you happen to know if the brightening of Sc is due mostly to the reduction of gamma or to the changes to the microphysics parameters?

We should assess whether the changes to autoconversion change the aerosol indirect effect. We don't want the AIE to be too strong. Andrew knows how to run simple tests.

[bstephens82]

Do you happen to know if the brightening of Sc is due mostly to the reduction of gamma or to the changes to the microphysics parameters?

I think both changes have the effect of brightening the Sc, although I am not sure which is dominant.

We should assess whether the changes to autoconversion change the aerosol indirect effect. We don't want the AIE to be too strong. Andrew knows how to run simple tests.

I can reach out to Andrew about this.

[bstephens82]

A couple of test I did on Friday look fairly decent I would say compared to my control case. These are still a bit too bright in the tropical EP and west of the South American coast in that region, but I would say things are looking at lot better off the coast of California. In both of these cases, I've adjusted micro_mg_dcs and micro_mg_autocon_fact on the microphys side and gamma_coef on the CLUBB side.

Here are diagnostics along with comparisons to zt0 (control run).
zt20:
https://webext.cgd.ucar.edu/F2000climo/zt20/atm/
(Profile plots can be found here: https://webext.cgd.ucar.edu/F2000climo/zt20/atm/zt20_ANN/CLUBB_diagnostics.html)

zt21:
https://webext.cgd.ucar.edu/F2000climo/zt21/atm/

I noticed that zt20 also looks a little better with respect to surface stress:

[bstephens82]

@JulioTBacmeister and I looked at this recent test I did, which has some nice features,
https://webext.cgd.ucar.edu/F2000climo/zt20/atm/zt20-obs.0001_1/set5_6/set5_ANN_SWCF_CERES-EBAF_obsc.png

and we were wondering about ways to break up the Sc near the ITCZ, since the Sc are coming too close to the ITCZ.  How can we shift these poleward? It could be that subsidence is not strong enough on the edges of the ITCZ, I'll check into that. We are also wondering if there is a correlation between surface heat flux and inversion strength at cloud top. @vlarson do you have any thoughts or suggestions about this?

[bstephens82]

Nice job, Ben. This is an encouraging start.

• On your Google Sheets list of runs, could you please indicate the parameter values and flag settings? E.g., perhaps you could link to (or excerpt) a log file for the default values and then for the modified runs in the list indicate the parameter values and flag settings that are different from the default. Also, on your list, could you please provide links to the plots for the important runs?

I'll work on including this information in the spreadsheet today. Once I have that taken care of I will link to the spreadsheet at the top of this github thread.

• For each location where a big error of interest occurs (e.g., near but not on the ITCZ), it might help to add an extra latitude-longitude pair to Zhun's plotting script (E3SM_CLUBB_diag.py). Then the script will create an extra profile plot at that location.

Okay I can try that, good idea. Might help determine what's happening near the ITCZ.

• With which model configuration did the near-ITCZ regions become so bright? Did that error first appear when l_diag_Lscale_from_tau was flipped from false to true? More generally, can you find a configuration of CAM that is similar to zt20 but that doesn't have the bright near-ITCZ regions, e.g., by rolling back some CLUBB and/or PUMAS flag settings or by using an older version of CAM that has older versions of CLUBB or PUMAS? If so, that might provide a useful benchmark. We could use it to compare plots and see which fields look different in zt20.

I have been trying to determine this. I ran a number of tests switching on and off the flags we are using for these runs, and unfortunately it is not clear so far which flag (if any) is responsible for the excessively bright ITCZ. The taus code alone does not suddenly cause a very bright ITCZ. Neither does the prognostic momentum flux flag. I have tried turning both of these on, and then switching on other flags, but have not yet figured out where the brightness is coming from. It may be some collection of flags that does it rather than a single flag, which is what I was afraid of. I'll review what I've tried here and give a more detailed description.

• I don't know if this is related to the excessive brightness near the ITCZ, but in many regions across the globe, wp3 < 0 near the lower surface. This seems wrong. The negative wp3 might be related to the negative spike of wp3_ta (i.e., -dwp4/dz) at the lower surface. This, in turn, seems related to the large value of wp4 right at the surface. I wonder if this is the result of a large value of gamma_coef. I'm not sure. At UWM we often run with a lower value of C8 and a lower value of gamma_coef.

Where are you seeing this with wp3? In the zt20 profile plots I linked above (if so I think I know what you mean)? Yes I've noticed that some of the parameter values are pretty different from what we typically ran with at UWM, and I've wondered about the impact of those large numbers, thanks for your thoughts here.

[bstephens82]

I looked a little more closely at this negative wp3 question, and from the budgets it looks like what might be happening is that the wp4 is too strongly sloped near the surface, which gives a large negative wp3_ta term. The tp term typically balances the ta term, but in this case it seems it's not able to balance it out. The large wp4 slope in turn seems to come from wp2 being compressed near the surface and hence more positively sloped as well. So one way to stop wp3 from going negative might be to boost wp2 higher into the atmosphere somehow. So maybe we could try reducing the value of terms that offset wp2 (c_uu_shr, c_uu_buoy, and relevant tau parameters).

Default 092 tag (no taus, no PMF, etc.)

Our new setup with taus, PMF, etc.:

[adamrher]

Is it possible that the b.c.'s for w'2 are systemically underestimated, leading to too steep a gradient across the first model level? Looking into the code, the b.c.'s are a combination of ustar and wstar ... l_vary_convect_depth will impact wstar. Is that on in these simulations?

I also wonder if it's worth coming up with a more accurate ustar estimate. It currently just takes the squared surface stresses raised to the 1/4 power ... in the long term I think we should start forking this in from the CPL, but in near term maybe we can improve upon the estimate by using the surface winds to back ustar out of the surface stresses?

[megandevlan]

Also probably worth noting that when l_vary_convect_depth is off, the wstar definition used here depends on z_const. In the surface module of CLUBB, z_const = 1 m, and that seems like it could be unrealistically small. In all the CLM-based calculations I've looked, it seems like ZBOT is used rather than a constant of ~1m. If this is making wstar unrealistically small, maybe that's also contributing to wp2 being underestimated.

[bstephens82]

Thanks for your thoughts Adam and Meg, l_vary_convect_depth is normally true in the runs I'm doing. I will do a few tests to see what effect that is having. I probably get too focused on the tunable parameter values.

[bstephens82]

I used the DiagnosticsAndTools plotting code to plot out three cases from the 092 tag:

1. "dflt092" -- out of the box cam_dev L58 run
2. "d92taus" -- cam_dev L58 run with taus code turned on using Colin's parameters
3. "d92taus_zhun_c5zero_e3sm_flags2" --- same as (2) except I'm using Zhun Guo's parameters from his 2021 paper on the taus code, or something very close to it.

The new profile plots are here. I put a grid of I think 40 points over the region that shows the persistent bright spot off of western South America, so we could get a good look at diagnostics in this area. These points are labeled "WSA" on the page that opens when you click the link for the plots.

I'm still looking over these plots, but one thing that sticks out to me right away are the differences in the skewness functions.

[adamrher]

Lots to digest. Option (3) really messes up dycoms. It produces very large skewness and very anemic Sc clouds at the 900 hPa level. The latter applies to (2) as well.

[adamrher]

I'm seeing large differences in omega among the simulations. I'm wondering if there is an impact on tropical variability. Ben can you point me to these runs and I'll make some wheeler-kiladis plots?

[bstephens82]

Sure these tests are all at /glade/scratch/stepheba/archive/X where X is the name of the test above (dflt092, etc.).

[bstephens82]

I'm seeing large differences in omega among the simulations. I'm wondering if there is an impact on tropical variability. Ben can you point me to these runs and I'll make some wheeler-kiladis plots?

Zhun's tuning was originally used for E3SM, just FYI. I don't know offhand what might be expected to change when you apply an E3SM tuning directly to CAM.