Need to get FATES working for NEON

[ekluzek]

We need to make it easy to run with FATES for a NEON site. Currently there isn't a compset alias for FATES with tower data. So we need to add that. Second, the user mods for FATES currently assume that BGC is active. So for example ndep and fire data are set in the namelist, things that FATES normally either doesn't use or only under certain conditions.

[ekluzek]

I see two ways to accomplish this. One would be to triple the number of user mod directories for NEON and have a base, BGC, and FATES version. The base version could also be used for SP simulations but would be included in the BGC and FATES versions.

The other way would be to add more attributes in the namelist XML files that operate on CLM USRDAT NAME and do something different if NEON.

I think the second is the best way to do it. That will embed NEON deeper into CTSM but I think that's ok. It won't have as much duplication as the first, and I think adding multiple types of user mod directories would be confusing for NEON.

[wwieder]

I agree here Erik, option 2 makes more sense to me to avoid duplication, but I'm not completely clear what's involved to ' add more attributes in the namelist XML files that operate on CLM USRDAT NAME'? If this is pretty straight forward, I'd say we move ahead on this.

[ekluzek]

The compset I think we should add is I1PtClm51FatesRs alias for longname: 2000_DATM%1PT_CLM51%FATES_SICE_SOCN_SROF_SGLC_SWAV

@wwieder what this would look like in the XML are statements like this:

<stream_year_first_ndep phys="clm5_1"  use_cn=".true." sim_year_range="1850-2100" clm_usr_name="NEON" >2018</stream_year_first_ndep>

<stream_fldfilename_ndep phys="clm5_1"  hgrid="0.9x1.25"   use_cn=".true."   ssp_rcp="SSP3-7.0" clm_usr_name="NEON"
>lnd/clm2/ndepdata/fndep_clm_f09_g17.CMIP6-SSP3-7.0-WACCM_2018-2030_monthly_c210826.nc</stream_fldfilename_ndep>

The perl code also has clm_usr_name attribute added to it so that the following:

    add_default($opts, $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, 'stream_fldfilename_ndep', 'phys'=>$nl_flags->{'phys'},
                'use_cn'=>$nl_flags->{'use_cn'}, 'lnd_tuning_mode'=>$nl_flags->{'lnd_tuning_mode'},
                'hgrid'=>"0.9x1.25", 'ssp_rcp'=>$nl_flags->{'ssp_rcp'}, 'nofail'=>1 );

Gets changed to:

    add_default($opts, $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, 'stream_fldfilename_ndep', 'phys'=>$nl_flags->{'phys'},
                'use_cn'=>$nl_flags->{'use_cn'}, 'lnd_tuning_mode'=>$nl_flags->{'lnd_tuning_mode'},
                'hgrid'=>"0.9x1.25", 'ssp_rcp'=>$nl_flags->{'ssp_rcp'}, 'nofail'=>1 , clm_usr_name=$opts->{'clm_usr_name'};

I also think we should introduce a build-namelist use-case for NEON rather then just using the SSP3-7.0 SSP use-case for everything in CLM. This would move some of what's in the user-mod directory into the use-case for NEON which handles it at a higher level. Technically everything could be moved into the use-case for NEON, but I think we should retain user-mods to make NEON look similar to how things work when the user creates a case for their own tower site. But, there are some special settings that are NEON specific that don't look like other cases. These are the type of things that should be put into the NEON use-case.

[ekluzek]

The bit about NEON use-cases is in #1363. It suggests to create compsets that are specific to 2018 and 2018 to present day, I'm not if we want such compsets, but we should definitely do use-cases for it.

[wwieder]

I think the compset makes sense but have a few question:

• nothing in the compset long name makes me think this will be a NEON case. Can users still run a case for a generic point that uses this compset or do the XML changes get set in the usermods directory? (sorry for these basic questions, I've spent little time retaining understanding the mechanics of what going on under the hood as we're running NEON cases).
• I though FATES didn't need NDEP data?
• If FATES does need NDEP, should it also point to the clipped ndep file that Jim created to reduce download times in the container?
• Down the road @adrifoster is hoping to generate data to initialize the model from NEON AOP data. Where does this get brought into the XML or other changes being suggested here?

[adrifoster]

• nothing in the compset long name makes me think this will be a NEON case. Can users still run a case for a generic point that uses this compset or do the XML changes get set in the usermods directory? (sorry for these basic questions, I've spent little time retaining understanding the mechanics of what going on under the hood as we're running NEON cases)

This compset is not unique to NEON, and I don't believe it should be. We are just specifying the component sets to run FATES at any single point here. So yes this is what users should use for a generic single-point FATES run (data-driven atmosphere, stub ice, etc.)

Down the road @adrifoster is hoping to generate data to initialize the model from NEON AOP data. Where does this get brought into the XML or other changes being suggested here?

This currently would require an addition of user_nl_clm namelist variables: use_fates_inventory_init = .true. and fates_inventory_ctrl_filename= with the correct files and paths. I'm not sure these need to be XML variables, but I don't have strong opinions either way.

[ekluzek]

Yes, exactly the compset name and alias can be used for a generic tower site to run FATES with it. We currently don't have any NEON specific compsets. And because we also have NEON user-mod directories I think it should stay that way. The only thing that the user needs to keep straight are to use either the BGC single point compset or the FATES single point compsets and use the NEON user-mod directory.

Yes, FATES doesn't need NDEP data and that's actually the problem. Because the user-mod directory for NEON specifically says what NDEP file to use -- you have to remove that line by hand (or we have to have multiple user-mod directory levels). Or we have to do what I suggest in the XML. The advantage of the XML is that it can figure out that you don't need the NDEP files for FATES. It can also figure out if you need the lightning and pop-density files which you sometimes do in FATES if you turn on certain spitfire options.

I think to set the finidat files for NEON we should add them to the NEON specific use case. We'll leave the fsurdat files in the user-mod directory because that makes it look like the generic tower site case. Technically we could put the finidat files in either the user-mod directory or in the XML, but I suggest we put it in the XML. The XML allows for more complex handling and options, and doesn't cause problems if the user modifies the user_nl_clm file for a case for their own customization. I might need to explain that last point in person.

If @adrifoster is also going to prepare inventory files for FATES I still think the fates_inventory_ctrl_filenames should be put in the use-case XML file. I can show people what this means more tomorrow.

Also because we use XML in many different places in the code, sometimes it's unclear. In this context I'm mainly talking about adding a NEON use-case which is a new file under bld/namelist_files/use_cases. I'm thinking something like

2018_NEON_control.xml

and/or

2018-PD_NEON_transient.xml

Sometimes when we talk about adding an XML variable verses a user-mod directory change we mean adding an XML variable to the case directory in env_run.xml for example. In this case I'm not suggesting that, only adding a new XML file under the use_cases directory that is NEON specific.

[ekluzek]

Some of these details are only going to be seen by us developers. In the end I suggest that what the user sees would be something like these three cases:

./create_newcase --res CLM_USRDAT --compset I1PtClm51Bgc --user-mods-dir NEON/YELL --case <CASEDIR>
./create_newcase --res CLM_USRDAT --compset I1PtClm51Fates --user-mods-dir NEON/YELL --case <CASEDIR>
./create_newcase --res CLM_USRDAT --compset IHist1PtClm51Bgc --user-mods-dir NEON/YELL --case <CASEDIR>

The top two would be used for spinup. The last one could be used for transient 2018-2020 cases (that's the place that it might have a good mechanism to shift between a spinup case and a transient case). If there is a need for a FATES transient case we could also add that compset as well.

[wwieder]

Thanks for the clarification here.

One more question: why do we need a different compset for a spinup vs. transient simulations I1PtClm51Bgc vs IHist1PtClm51Bgc? I can see this subtle distinction being confusing to users.

I'm assuming this has different defaults for datm_streams files and how certain xml variables are defined in env_run? OR, do the different compsets just define different settings that are called in the --user-mods-dir?

[ekluzek]

OK, we had a lively discussion on this at the CTSM software meeting this morning! My opinion went back and forth on this during the discussion. But, I feel like I have a clear idea of what to do now. And I think the way forward is for me to create a little PR that does what I envision and then we can discuss based on what it looks like. I think it would help to have a concrete example to look at.

One thing that @negin513 brought out that I had forgotten in the discussion above is that you CAN put if logic in the shell_commands. This is a little tricky, with some limitations, and downsides, and I would say it's better to put more complex logic into the use-cases or namelist defaults. But, we should think about when do you put something in each place: compset, use-case, user-mods, namelist-defaults, or offline run script.

That said there were several good design goals that came out of the discussion that's worth keeping in mind. Please help me to establish these as best we can define them.

• Have as few compsets as possible for single-point and use the same compset for generic tower sites as well as NEON.
• Have the process for running NEON as similar to generic tower sites as possible
• It's OK for I-only cases to have compsets that don't encapsulate the exact year being simulated
• In contrast higher level compsets (B, E, F, etc.) do always keep the exact year in the name
• It's OK for single-point cases to use the same compset for both spinup and transient simulation in contrast to global simulations where we separate the two

I feel like those two came directly out the discussion. But, I'd like to add a few that we will need to discuss and make sure we all agree on.

• Avoid putting complex logic in user-mod shell_commands
• Avoid putting things in user-mods that can be handled in use-cases or namelist-defaults
• Avoid putting things in use-cases that can be handled in namelist-defaults
• Avoid putting customization in run-scripts that can be handled in the above infrastructure as that allows people who don't use the run-scripts to have access to it as well.

The bottom line for above is that we'll have three compsets used for single-point cases. One for SP, one for BGC and one for FATES, they'll all be for fixed present-day conditions. Things that would be in the normal user-mod directories for generic cases will be in there for NEON. The NEON use case will be used for some of the changes that are specific to NEON. We'll try to favor putting NEON customization in namelist-defaults rather than the NEON use-case.

[ekluzek]

@adrifoster pointed out this problem and I thought she could get around it, with use_crop=F, but issue #1505 shows that you can't do that. So in some ways #1505 is a blocker for this issue.

[negin513]

Okay, to summarize from @wwieder : To run FATES, we have to generate the surface dataset using 16 pfts for FATES. So I am linking this to #1461 to this, to clarify what needs to be done for subset_data for FATES.

[ekluzek]

@negin513 we should be able to change the model so that it can run single-point simulations with FATES for 78 PFT surface datasets. It would be a pain to have to support both the 78 PFT surface datasets and the 16pft ones. Especially, since FATES actually ignores the surface dataset anyway. So requiring that we create 16 pft datasets for FATES seems silly.

We should maybe talk about this more.

[wwieder]

Polly created this for single point cases, which could be helpful for creating usermod_dirs for FATES. We may also want to do with with the default (16 pft) surface dataset.

/glade/u/home/pbuotte/Earthshot/scripts/

[jkshuman]

Noting that these single-point case scripts are a modification of work by @adrifoster for NEON sites

[ekluzek]

@adrifoster completed this in a slightly different way than we envisioned previously. We have a separate set of NEON surface datasets with 16 mixed PFT's. This is actually a better way to run with FATES, so the best solution.