ctsm5.3 f19 spinup and historical for PPE

[linniahawkins]

Description:
For the CLM6-PPE, we need a 2-deg f19 full grid cold start spinup.

We will start from these initial conditions and run our own spin-up protocol for each ensemble member.

For reference, this is the restart we used in the CLM5-PPE, but it's been moved and I don't know where it came from:
'/glade/p/cgd/tss/people/oleson/CLM5_restarts/clm51_PPEn02ctsm51d021_2deg_GSWP3V1_leafbiomassesai_PPE3_1850pAD.clm2.r.2041-01-01-00000.nc'

NOTE: CLM6-PPE is actually a ctsm5.3.0 PPE. It's okay if this spinup is done with a 5.2.02* tag, as long as it is close enough.

Timeframe: We would like to be able to start testing with the new restart by ~Oct.1 so we can kick off a full PPE by Oct. 15

---

Case directory:
Locally:
/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD
/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_SASU
/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_pSASU
/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_hist

---

Sandbox:
Locally:
/glade/work/oleson/ctsm5.3.0

---

Case modifications

AD:
user_nl_clm

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange MOSART_MODE=NULL
./xmlchange STOP_OPTION=nyears
./xmlchange STOP_N=40
./xmlchange REST_N=10
./xmlchange DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE
./xmlchange CLM_ACCELERATED_SPINUP=on
./xmlchange CLM_FORCE_COLDSTART=on
./xmlchange RESUBMIT=6

---

SASU, created from clone of AD case
user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_AD.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

hist_fincl1 = 'TOTECOSYSC', 'TOTECOSYSN', 'TOTSOMC', 'TOTSOMN', 'TOTVEGC', 'TOTVEGN', 'TLAI', 'GPP', 'CPOOL', 'NPP', 'TWS','H2OSNO'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange RUN_TYPE=hybrid
./xmlchange CLM_ACCELERATED_SPINUP=sasu
./xmlchange CLM_FORCE_COLDSTART=off
./xmlchange RESUBMIT=6

---

pSASU, created from clone of pSASU case
user_nl_clm

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_SASU.clm2.r.0281-01-01-00000.nc'
reseed_dead_plants = .true.

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

xml changes

./xmlchange CLM_ACCELERATED_SPINUP=off
./xmlchange RESUBMIT=3

---

hist:

finidat = '/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc'

! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'
flanduse_timeseries = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'

! turn off shifting cultivation
do_grossunrep = .false.

ndep_taxmode = 'extend'

hist_fincl1 += 'FSDSND','FSDSVD','FSDSNI','FSDSVI','RAIN_FROM_ATM','SNOW_FROM_ATM','FLDS_NOT_DOWNSCALED','ZBOT','Tair_from_atm','Thair_from_atm','QBOT_NOT_DOWNSCALED','PBOT_NOT_DOWNSCALED','Rho_from_atm','PCO2','UWIND','VWIND','BCPHIDRY','BCPHODRY','BCPHIWET','OCPHIDRY','OCPHODRY','OCPHIWET','DSTWET1','DSTDRY1','DSTWET2','DSTDRY2','DSTWET3','DSTDRY3','DSTWET4','DSTDRY4','ATM_TOPO'

use_init_interp = .true.

! Generated from ./create_newcase ..... --user-mods-dir output_crop_highfreq and edited for only h0,h1,h2,h3,h4

!----------------------------------------------------------------------------------
! Settings from output_base
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
! h0 stream ICE variables and other variables inactive by default (monthly average, gridcell-level)
! Emon, Lmon, LIMon
hist_mfilt(1) = 1
hist_dov2xy(1) = .true.
hist_nhtfrq(1) = 0
hist_type1d_pertape(1) = ' '
!hist_fexcl1 += 'PCT_GLC_MEC','PCT_NAT_PFT'
hist_fexcl1 += 'PCT_GLC_MEC'
hist_fincl1 += 'EFLX_LH_TOT_ICE', 'FIRE_ICE', 'FLDS_ICE', 'FSH_ICE', 'FSNO_ICE', 'FSR_ICE', 'QFLX_SOLIDEVAP_FROM_TOP_LAYER_ICE', 'QRUNOFF_ICE', 'QSNOFRZ_ICE', 'QSNOMELT_ICE', 'RAIN_ICE', 'SNOW_ICE', 'SNOWICE_ICE', 'SNOWLIQ_ICE', 'SNOTXMASS_ICE', 'TG_ICE', 'TOPO_COL_ICE', 'TSA_ICE', 'TSOI_ICE'

! h1 stream (monthly average, finest sub-grid)
! Emon, Lmon
hist_mfilt(2) = 1
hist_dov2xy(2) = .false.
hist_nhtfrq(2) = 0
hist_type1d_pertape(2) = ' '
hist_fincl2 += 'TLAI', 'TSA','TREFMNAV','TREFMXAV', 'BTRANMN', 'Vcmx25Z', 'FSH', 'VEGWP', 'FCTR', 'FCEV', 'FGEV', 'FIRE', 'FSR', 'FIRA', 'FSA', 'GSSUNLN', 'GSSHALN', 'TSKIN'

! h2 stream (monthly average, landunit-level)
! Emon
hist_mfilt(3) = 1
hist_dov2xy(3) = .false.
hist_nhtfrq(3) = 0
hist_type1d_pertape(3) = 'LAND'
hist_fincl3 += 'FSR', 'H2OSNO', 'Q2M', 'SNOWDP', 'TSA', 'TREFMNAV', 'TREFMXAV', 'TG', 'QRUNOFF', 'FSH', 'FIRE', 'FIRA', 'FGR', 'EFLX_LH_TOT', 'RH2M', 'TLAI', 'SOILWATER_10CM', 'TOTSOILLIQ', 'TOTSOILICE', 'U10', 'TSOI_10CM', 'QIRRIG_DRIP', 'URBAN_HEAT', 'WASTEHEAT', 'TSKIN'

! h3 stream (yearly average, gridcell-level)
! Eyr
hist_mfilt(4) = 1
hist_dov2xy(4) = .true.
hist_nhtfrq(4) = -8760
hist_type1d_pertape(4) = ' '
hist_fincl4 += 'PCT_GLC_MEC', 'QICE_FORC', 'TSRF_FORC', 'TOPO_FORC', 'PCT_NAT_PFT','PCT_LANDUNIT','FSNO_ICE'

! h4 stream (yearly average, landunit-level)
! Eyr landuse tiles
! (No fields in base, but other usermods add to this stream)
hist_mfilt(5) = 1
hist_dov2xy(5) = .false.
hist_nhtfrq(5) = -8760
hist_type1d_pertape(5) = 'LAND'

!----------------------------------------------------------------------------------
! Settings from output_bgc
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'SOM_ACT_C_vr', 'SOM_ACT_N_vr', 'SOM_SLO_C_vr', 'SOM_SLO_N_vr', 'SOM_PAS_C_vr', 'SOM_PAS_N_vr', 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_C_vr', 'CWD_N_vr', 'SMIN_NO3_vr', 'CONC_O2_UNSAT', 'CONC_O2_SAT','SMIN_NH4_vr','SMINN_vr'
hist_fincl1 += 'LEAFC_TO_LITTER', 'FROOTC_TO_LITTER','LIT_MET_C_TO_SOM_ACT_C','LIT_MET_N_TO_SOM_ACT_N','LIT_CEL_C_TO_SOM_ACT_C', 'LIT_CEL_N_TO_SOM_ACT_N','LIT_LIG_C_TO_SOM_SLO_C','LIT_LIG_N_TO_SOM_SLO_N','DWT_WOOD_PRODUCTC_GAIN_PATCH'

! h1 stream (monthly average, finest sub-grid)
hist_fincl2 += 'GPP', 'NPP', 'AGNPP', 'TOTVEGC', 'NPP_NUPTAKE', 'AR', 'HR', 'HTOP'

! h2 stream (monthly average, landunit-level)
! TOT_WOODPRODC:I, and NEE are not available at the landunit level
hist_fincl3 += 'GPP', 'NPP', 'AR', 'HR', 'DWT_CONV_CFLUX_PATCH', 'WOOD_HARVESTC', 'DWT_WOOD_PRODUCTC_GAIN_PATCH', 'SLASH_HARVESTC', 'COL_FIRE_CLOSS', 'DWT_SLASH_CFLUX_PATCH', 'FROOTC:I', 'HTOP'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr', 'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_N_vr','CWD_N_vr', 'TOTLITC:I', 'TOT_WOODPRODC:I', 'TOTSOMC:I','TOTVEGC:I'

! h4 stream (yearly average, landunit-level)
hist_fincl5 += 'TOTSOMC:I', 'TOTSOMC_1m:I', 'TOTECOSYSC:I', 'TOTVEGC:I', 'WOODC:I', 'TOTLITC:I', 'LIVECROOTC:I', 'DEADCROOTC:I', 'FROOTC:I'

!----------------------------------------------------------------------------------
! Settings from output_crop
!----------------------------------------------------------------------------------

! h0 stream (monthly average, gridcell-level)
hist_fexcl1 += 'PCT_CFT'

! h3 stream (yearly average, gridcell-level)
hist_fincl4 += 'PCT_CFT'

user_nl_datm_streams:

CLMCRUNCEPv7.Solar:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc, ...through 2023
CLMCRUNCEPv7.Solar:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.Solar:datavars= FSDS Faxa_swdn

CLMCRUNCEPv7.Precip:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.Prec.1901.nc, ...through 2023
CLMCRUNCEPv7.Precip:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc

CLMCRUNCEPv7.TPQW:datafiles=/glade/derecho/scratch/wwieder/TRENDY2024/inputs/three_stream/clmforc.CRUJRAv2.5_0.5x0.5.TPQWL.1901.nc, ...through 2023
CLMCRUNCEPv7.TPQW:meshfile=/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream/mesh_cdf5.nc
CLMCRUNCEPv7.TPQW:datavars = QBOT Sa_shum,PSRF Sa_pbot,TBOT Sa_tbot,WIND Sa_wind,FLDS Faxa_lwdn

---

SourceMods:

/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD/SourceMods/src.clm/CanopyFluxesMod.F90 (sa_leaf = sa_leaf + esai; see PR#2788)

---

Diagnostics:
Diags (if available)

Standard:
[https://webext.cgd.ucar.edu/I1850/$CASE/lnd/](https://webext.cgd.ucar.edu/I20TR/ctsm530_f19_PPE_hist/lnd/ctsm530_f19_PPE_hist_2004_2023-ctsm53n04ctsm52028_f09_hist_2004_2023/setsIndex.html)
ILAMB:
https://webext.cgd.ucar.edu/I20TR/ctsm530_f19_PPE_hist/lnd/_build_ctsm530_f19_PPE_hist/

---

Output:
Output (if still available):
/glade/scratch/oleson/archive/ctsm530_f19_PPE_AD/
/glade/scratch/oleson/archive/ctsm530_f19_PPE_SASU/
/glade/scratch/oleson/archive/ctsm530_f19_PPE_pSASU/
/glade/scratch/oleson/archive/ctsm530_f19_PPE_hist/

Timeseries files:
/glade/derecho/scratch/oleson/timeseries/ctsm530_f19_PPE_hist/lnd/proc/tseries

---

Contacts:
@linniahawkins
@olyson

---

Extra details:
AD: 280 yrs, 2.08% disequilibrium
SASU: 280 yrs, 13.73% disequilibrium
pSASU: 160 yrs, 2.15% disequilibrium
hist: 1850-2023

[wwieder]

Thanks for creating this issue, Linnia. I'd suggest we use the same set up as in #69, but with f19. Maybe it makes sense for us to look at 69 first (at least in 1850) to make sure we don't need any quick fixes or additional adjustments?

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

[slevis-lmwg]

Alternatively we can at least do this in the ctsm5.3 code base when it's tagged next week? What do you think @slevis-lmwg and @olyson?

At the moment either seems fine, since #69 is using the latest ctsm5.3 tag (branch_tags/ctsm5.3.n04_ctsm5.2.028), though we should first have resolution regarding Keith's latest posts in issue 2567.

[olyson]

The f09 and ne30 spinups should be in their post-SASU phase by this afternoon, so we could probably start looking at results tomorrow afternoon. As I posted in ESCOMP/CTSM#2567, the differences in CNVegStructMod.F90 code were bfb, and the changes to sa_leaf in CanopyFluxesMod.F90 resulted in small differences, at least in I2000 cases as I posted in ESCOMP/CTSM#2777.

[wwieder]

Is it worth touching base on this at some point on Thursday? 930 or 100 times are better for me

[olyson]

Looks like we have a CTSM5.3 tag now, so it seems like we could start this. Could include the sa_leaf change as a SourceMod. One question I have is that our approach last time was to run without the crop model on since we weren't changing crop parameters. But we weren't tuning either. So I assume we'll run with the crop model on...? I can meet Thurs, but we have CTSM software meeting at 9:30?

[wwieder]

Good idea on the sa_leaf issue, @olyson. I'll let @linniahawkins or @djk2120 weigh in here if they want the crop model on for the 2 degree spinup?

[olyson]

A couple of other questions:

1. No need to have isotopes on right?
2. We have been setting do_grossunrep = .false. (turn off shifting cultivation). Do the same here?

[wwieder]

@slevis-lmwg let's make a 16PFT surface dataset at the f19 resolution in 1850 + a historical simulation.

[slevis-lmwg]

I submitted the job to generate new fsurdat/landuse files earlier this afternoon.
(Issue ESCOMP/CTSM#2791)

[olyson]

Crops off (use 16pft dataset).

[slevis-lmwg]

The files are ready (again, details in ESCOMP/CTSM#2791)

[olyson]

I think this is ready to start (beginning with the AD). But it might be good for anyone who is interested to check the setup:
/glade/work/oleson/ctsm5.3.0/cime/scripts/ctsm530_f19_PPE_AD
@wwieder @slevis-lmwg @linniahawkins @djk2120

[djk2120]

everything matches my expectations

[djk2120]

is it still appropriate to include: reseed_dead_plants = .true. edit: I can see that this is indeed the setting

[slevis-lmwg]

My understanding is: yes.

[slevis-lmwg]

Looks good to me.

[olyson]

reseed_dead_plants is set to .true. by default, but in a cold start (as this AD is) I don't think it actually does anything.

[olyson]

This spinup is complete. The restart file is:

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

I'm running a branch now to look at survivability. Pending that looking ok, I plan to proceed with the historical.

[olyson]

[linniahawkins]

Thanks Everyone! I am testing this restart in our PPE workflow now.
@olyson can you change the permissions on this file: /glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

[olyson]

Ok, done.

[djk2120]

@olyson @wwieder Would it be possible to continue this simulation through the full historical? It would be really helpful for assessing if our old sparsegrid still works, and for training a new sparsegrid if not.

[wwieder]

Seems reasonable. Can you kick off a historical run, Keith?

[olyson]

Yes, I'll start that up.

[olyson]

The historical is done. The output is here:

/glade/derecho/scratch/oleson/archive/ctsm530_f19_PPE_hist

I'll run some diagnostics when I'm back in the office.

[linniahawkins]

Great, thank you Keith!

[olyson]

Survivability and max TLAI from the historical:

[wwieder]

@olyson when you are able, can you create an SSP2-4.5 continuation with this case through 2100? (I think that's the land use timeseries we have available for the f19 resolution with 5.3 datasets?). I'll do the same with #72

[olyson]

It looks like the one to use is:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240905.nc

or:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_ne0np4.ARCTIC.ne30x4_SSP2-4.5_1979-2026_78pfts_c240908.nc

The historical landuse file that was used for the PPE historical was:

/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc

@slevis-lmwg , do you have a recommendation for which file to use, thanks.

[slevis-lmwg]

Though you didn't list it in your question, I recommend this one:
landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240908.nc

All the files dated 0905.nc are an earlier iteration.
@wwieder @olyson @ekluzek how about I remove them now to prevent recurring confusion?

[olyson]

Sorry, copy/pasted the wrong one with the 240908 date. That makes sense.

[wwieder]

These are no crop runs is the 78 pft data ok, or should we use 'glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'?

[slevis-lmwg]

You could, though the 78pft file is fine, too. Linnia is using the 16pft file to run a little faster if I remember correctly.

[olyson]

That's a historical that only goes through 2023, right? I used it for the PPE historical. I was thinking about using:

fsurdat = '/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_1.9x2.5_hist_1850_78pfts_c240908.nc'
flanduse_timeseries = '/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_SSP2-4.5_1850-2100_78pfts_c240908.nc'

[olyson]

@wwieder , I think we want the anomaly forcing settings below. Otherwise, since we are starting in 2024, the datm will get the last time slice of the data (Dec 2100) of the anomaly forcing to interpolate from, at the start of the simulation:

Anomaly.Forcing.Temperature:year_first=2023
Anomaly.Forcing.Temperature:year_last=2100
Anomaly.Forcing.Temperature:year_align=2023

[wwieder]

sounds good

[olyson]

The SSP245 anomaly simulation is complete. Do we need to run some diagnostics?

[wwieder]

let's not worry about it now. I did some spot checking of results of fields Tom may be interested in, but he's likely going to want to make nicer figures for a publication / presentation