stability with spectral element dynamical core
Welcome to the CAM wiki!
Keeping everything else the same (i.e. only add HB scheme where CLUBB is not active) using coupled_eval3 code base (tag 6_3_110) and tunings (approximately) 11-month averaged (1 month spinup) OMEGA500 field is mostly positively affected (upper is reference; lower is HB scheme turned on where CLUBB is not active)
Contour: [-0.2,0.2]
Code base: https://github.com/PeterHjortLauritzen/CAM/tree/clubb_hb with May 12 hash 28bdd26590b364dc8a29cc25bb5c616df802e6da
HB on and:
se_nu_p = 1E15
se_nu = 1E15
se_nu_div = 1E15
se_sponge_del4_nu_div_fac = 4
Same plot as above but only 3 months average:
Looks like up and down-drafts are degraded (too large!)
Abandon this option although it would have been nice to damp all modes the same ...
HB on and:
2x CSLAM time-step
se_nsplit = 1
se_rsplit = 6
Only changing this is unstable in sponge.
Increase sponge layer damping?
se_nsplit = 1
se_rsplit = 6
se_hypervis_subcycle_sponge = 2
se_nu_top = 5E5
is stable but even after one month spinup OEMGA500 is noisy (plot shows 1 month average of OMEGA500 for second month of simulation)
This result was expected since se_ftype=1 had the same noise (all the physics tendencies with nsplit=1 and se_ftypte=0 are added in one tendency - no dribbling!).
70-layers stable setting (16 day test after initial spinup):
se_sponge_del4_lev = 6
se_hypervis_subcycle = 2
se_nsplit = 5
Initial condition file (spun up for 16 days):
/glade/p/cgd/amp/pel/inic/FW2000.ne30pg3_ne30pg3_nlev70.i.nc
These namelist settings give the following stability estimates:
Estimates for maximum stable and actual time-steps for different aspects of algorithm:
(assume max wind is 400.00000000m/s)
(assume max gravity wave speed is 342m/s)
* dt_dyn (time-stepping dycore ; u,v,T,dM) < 304.38s 90.00s
* dt_dyn_vis (hyperviscosity) ; u,v,T,dM) < 45.30s 45.00s
* dt_tracer_se (time-stepping tracers ; q ) < 92.67s 90.00s
* dt_tracer_vis (hyperviscosity tracers; q ) < 339.77s 90.00s
* dt_tracer_fvm (time-stepping tracers ; q ) < 555.99s 360.00
* dt_remap (vertical remap dt) 360.00
* dt (del2 sponge ; u,v,T,dM) < 567.55s 90.00s
* dt (del2 sponge ; u,v,T,dM) < 1188.65s 90.00s
* dt (del2 sponge ; u,v,T,dM) < 2876.98s 90.00s
* dt (del2 sponge ; u,v,T,dM) < 7466.28s 90.00s
* dt (del2 sponge ; u,v,T,dM) < 19941.29s 90.00s
Spin-up setting:
se_sponge_del4_lev = 10
se_nu_div=3E16
se_sponge_del4_nu_fac = 5
se_sponge_del4_nu_div_fac = 8
se_hypervis_subcycle = 4
se_nsplit = 10
What does not work:
- Rayleigh friction at model top does not seem to help at all
- dadadj_nlvdry: dry convective adjustment does not seem so do much for max winds in the sponge
What seems to work:
- Turning on HB eddy scheme (unfortunately CLUBB is on too): source mods here: src/for-hb-hack/
- Spin-up efficiently: Setting se_nsplit =10 and se_nu_div = 3E16 keeps max winds under control (see plot below); I ran 16 day spinup - I think 1 day is enough based on Figure below.
Spin-up results:
Figure shows max wind in level 1 for different configurations. Red line is out of the box with se_nsplit=25; winds almost reach 800m/s and only slowly settle down. X-axis increments are 1.5 minutes so red line is a 1 day run. Green line is se_nsplit=10 and se_nu_div=3E16 (default is se_nu_div=2.5E15). After 1 day the winds are under 400 m/s and much more realistic.
Very hard to spinup. Start with nsplit=100 for 2 days, then 50 for 2 days, then 10 for 5 days and then 6 for 5 days until model was stable with:
se_sponge_del4_lev = 6
se_hypervis_subcycle = 2
se_hypervis_subcycle_sponge = 2
se_nu_top = 1E6
se_nsplit = 5
Time-series of max/min wind as a function of time-step. X-axis spans 15 days and started from a 2 days spinup using nplit=100. That last 5 days were run with the above settings.
Here is spunup initial condition:
/glade/p/cgd/amp/pel/inic/nlev110_FWsc2000climo_ne30pg3_ne30pg3.i.nc
I am spinning up initial conditions using the FWsc2000 compset which uses specified chemistry. If using a spun-up initial condition from an FWsc2000 run in a full chemistry run (FW2000) then the model hangs at the first time-step with many error messages from chemistry in cesm.log. My process has been to overwrite U,V,T,Q,PS in the non spun-up file with these variables from a FWsc2000 run:
ncks -m -A -C -v U,V,T,Q,PS spinup.ic.nc orig.ic.nc
Very hard to spin-up:
se_sponge_del4_lev = 30
se_nu_div=3E16
se_sponge_del4_nu_fac = 5
se_sponge_del4_nu_div_fac = 8
se_hypervis_subcycle = 4
se_nsplit = 200
U at level 1 after 0,30,60,90 minutes (note extremrly large unphysical winds triggered by imbalances between flow and orography):
Winds:
Crashes after this ... (even with se_nsplit=200).
Ran succesfully 1 day when turning on Rayleigh friction:
se_sponge_del4_lev = 30
se_nu_div=3E16
se_sponge_del4_nu_fac = 5
se_sponge_del4_nu_div_fac = 10
se_hypervis_subcycle = 4
se_nsplit = 100
rayk0=4
raykrange=2
raytau0=0.005
U at level 1 after 2h and 1 day:
Min/max winds for level 1 after 2h and 1 day:
Ran 4 days with:
se_sponge_del4_lev = 30
se_nu_div=3E16
se_sponge_del4_nu_fac = 5
se_sponge_del4_nu_div_fac = 10
se_hypervis_subcycle = 4
se_nsplit = 50
rayk0=4
raykrange=2
raytau0=0.01
Sasha ran a couple of years with b spun-up initial condition:
Sill getting spiles in wind close to 600m/s!
sponge_del4_nu_fac = 0.50E+01
sponge_del4_nu_div_fac = 0.75E+01
sponge_del4_lev = 10
se_hypervis_subcycle = 2
se_hypervis_subcycle_sponge = 1
se_hypervis_subcycle_q = 1
se_nu_top = 0.10E+07
se_rsplit = 4
* dt_dyn (time-stepping dycore ; u,v,T,dM) < 304.38s 90.00s
* dt_dyn_vis (hyperviscosity) ; u,v,T,dM) < 45.30s 45.00s
* dt_tracer_se (time-stepping tracers ; q ) < 92.67s 90.00s
* dt_tracer_vis (hyperviscosity tracers; q ) < 339.77s 90.00s
* dt_tracer_fvm (time-stepping tracers ; q ) < 555.99s 360.00
* dt_remap (vertical remap dt) 360.00
* dt (del2 sponge ; u,v,T,dM) < 258.42s 90.00s
* dt (del2 sponge ; u,v,T,dM) < 1293.75s 90.00s
Since we changes topography many of the IC files make the model unstable. Trying to copy PS from a spun-up simulation to the unstable IC file:
ncks -m -A -C -v PS ps.nc ic.nc
Run 6 hours:
se_statefreq = 50
se_nsplit = 50
se_sponge_del4_lev = 120
se_nu_p = 7.5E16
inithist='6-HOURLY'
With new IC file run 6 hours (could also have run previous setup 12 hours):
se_statefreq = 10
se_nsplit = 10
se_sponge_del4_lev = 120
se_nu_p = 7.5E16
se_hypervis_subcycle = 14
se_hypervis_subcycle_q = 2
inithist='6-HOURLY'
Setting se_nu_p = 7.5E16 seems to help at lot! That said, this created strong topo signature in OMEGA500.
Final spin-up used
se_sponge_del4_lev = 50
se_sponge_del4_lev = 5
se_sponge_del4_nu_div_fac = 2
se_statefreq = 20
se_nsplit = 20
se_nu = 1E16
with a hybrid PGF formulation (se_pgf_formulation=3).
After spin-up here are is max wind at level 1 and 16 for an 8 day run:
One thing I did not realize was that the namelist default sets:
<se_nu>-1</se_nu>
<se_nu waccmx="1">5.e15</se_nu>
<se_nu_div> -1 </se_nu_div>
<se_nu_div waccmx="1"> 10.e15 </se_nu_div>
<se_nu_p>-1 </se_nu_p>
<se_nu_top > 1.25e5 </se_nu_top>
<se_nu_top waccmx="1" > 1.0e6 </se_nu_top>
<se_nu_top model_top="mt" > 1.0e6 </se_nu_top>
se_nsplit=5
se_rsplit=3
se_qsplit=1
se_hypervis_subcycle = 3
se_hypervis_subcycle_q = 1
se_sponge_del4_lev = 6
se_nu_top=1e6
se_large_courant_incr = .false.
se_statefreq = 144
is unstable in long run.
Suggested setting (stable for 1 year):
se_sponge_del4_lev = 6
se_sponge_del4_nu_div_fac = 6
se_hypervis_subcycle = 2
se_hypervis_subcycle_q = 1
se_rsplit = 3
se_nsplit = 4
se_large_courant_incr = .false.
se_statefreq = 144
se_nu_top = 5E6
* dt_dyn (time-stepping dycore ; u,v,T,dM) < 570.71s 150.00s
* dt_dyn_vis (hyperviscosity) ; u,v,T,dM) < 86.73s 75.00s
* dt_tracer_se (time-stepping tracers ; q ) < 173.75s 150.00s
* dt_tracer_vis (hyperviscosity tracers; q ) < 520.36s 150.00s
* dt_tracer_fvm (time-stepping tracers ; q ) < 521.24s 450.00
* dt_remap (vertical remap dt) 450.00
* dt (del2 sponge ; u,v,T,dM) < 181.70s 150.00s
* dt (del2 sponge ; u,v,T,dM) < 909.67s 150.00s
Max/min winds in level 1 (top layer) as a function of time (1 year total):
To add file to our input-file server there are mandatory metadata that must be added to the file:
The mandatory metadata is:https://www2.cesm.ucar.edu/working_groups/Atmosphere/amwg_datasets.html
Also, please rename the file to contain _c230622 (or whatever date you update the file) and make any name adjustments (if needed).
Here is a script to change the global attributes (meta data):
#-data_title: Short title of the data
#-data_summary: Short paragraph about the data.
#-data_creator: your name and email
#-cesm_contact: Contact liaison or co-chair of AMWG
#-creation_date: Full date of dataset creation
ncatted -O -h -a data_title,global,c,c,"IC file for ne30L70 FW compset" 2.nc
ncatted -O -h -a data_summary,global,c,c,"This is an initial condition file spun-up for dynamic stability" 2.nc
ncatted -O -h -a data_creator,global,c,c,"Peter Hjort Lauritzen, pel@ucar.edu" 2.nc
ncatted -O -h -a cesm_contact,global,c,c,"Peter Hjort Lauritzen" 2.nc
ncatted -O -h -a creation_date,global,c,c,"09/06/2023" 2.nc
ncatted -O -h -a update_date,global,c,c,"06/06/2023" 2.nc
ncatted -O -h -a history,global,c,c,"None" 2.nc
ncatted -O -h -a data_script,global,c,c,"Unbalanced IC file spun-up using increased filtering and shorter dynamics time-steps; see guidance here: https://github.com/PeterHjortLauritzen/CAM/wiki/stability-with-spectral-element-dynamical-core" 2.nc