PLUMBER2 plumbing (csv file, wrapper script, usermods, and scripts).

[olyson]

Description of changes

This PR adds a csv file for PLUMBER2 tower sites in preparation for incorporating PLUMBER2 into the NEON framework.
The ncl script that generates this file is included as well

Specific notes

These are added to tools/site_and_regional for now.

Contributors other than yourself, if any: @ekluzek @wwieder @slevis-lmwg @TeaganKing @adrifoster

CTSM Issues Fixed (include github issue #):
Partial for #2137
Fixes #2414

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Testing performed, if any: None

[wwieder]

No changes need on my end...

[ekluzek]

This is exciting to see @olyson and @wwieder! There are some refactoring we'll do after the initial implementation comes in. There's a few things that I saw that we should do for this initial step.

Things done:

• Have usermod scripts update user_nl_datm_streams for each site and year start and end year for them [see this file]
• Decide on what orbital year should be update it see this file

These are not done and should be converted into issues (separate issues have line breaks between them):

• increase dtlimit for datm

• modify year start and year align on datm.streams to align with full calendar years

• remove o3 references in default/usernl_datm_stream

• WON'T FIX FOR NOW: update input data to cdf5 (may currently be netcdf4), nccopy -k cdf5

• Document new scripts and usage in the tools/site_and_regional/README file Documentation for Plumber2 refactoring #2409

• Move data to CESM inputdata and rimport it (me or @slevis-lmwg ) PLUMBER2 refactoring inputdata adjustments #2410

• Point to the new locations in usermod scripts
  (https://github.com/ESCOMP/CTSM/blob/8d3468597b0b9bb1db351f755c6662bb500e4b96/tools/site_and_regional/plumber2_usermods.py)

• Redo the users mods and update them

• Update some of the comments in the new scripts

• Erik -- add a unit test for subset_data for the latest addition PLUMBER 2 Refactoring - testing #2411

• Erik: Add a system test for a PLUMBER2 site US-Ha?

[ekluzek]

I added one more item to do before coming in, add a system test for a PLUMBER2 site, so we are at least testing one of the PLUMBER2 user-mods and are more likely to find issues that happen as the code changes.

[wwieder]

@ekluzek I could use some help with setting up datm.streams for the PLUMBER cases. I think we'd ideally use streams that span the hist-ssp timeframe as most start dates are before 2014 and a number of sites have end dates that end in 2017 (and eventually later). Maybe this is something we can discuss next week as you wrap your head around what else needs to happen here. Thanks

/glade/u/home/wwieder/scratch/PLUMBER2_tests/ZM-Mon_test2/user_nl_datm_streams contains a streamname 
'presndep.SSP3-7.0' that is not part of valid streamnames ['CLMGSWP3v1.Solar', 'CLMGSWP3v1.Precip', 
'CLMGSWP3v1.TPQW', 'presaero.SSP3-7.0', 'presndep.hist', 'preso3.hist', 'topo.observed', 'co2tseries.20tr'] 

[wwieder]

Incremental progress here.
my working case directory is still /glade/scratch/wwieder/PLUMBER2_tests/ZM-Mon_test2

Now I'm pointing to what should be the correct datm file:

<datafiles>
      <file>/glade/work/oleson/PLUMBER2/datm_files/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc</file>

But getting the following dt error that I don't know how to address?
@olyson how did you do this previously for PLUMBER2 sites?
(shr_stream_findBounds) ERROR: limit on and rDateIn lt rDatelvd
20071231.9166667 20080101.0000000
ERROR: (shr_stream_findBounds) ERROR: rDateIn lt rDatelvd limit true

[wwieder]

OK I can confirm that manually setting stream_dtlimit = 50._r8 in src/cpl/share_esmf/laiStreamMod.F90 allows the SP simulation to finish.

I think I've added information about lai_dtlimit to get is as a namelist variable that users can modify, but maybe someone can check my work? It's an open PR on Keith's PLUMBERcsv branch.

[wwieder]

I think this PR may be (almost) ready to merge. I've now run test spinup and historical cases with sp and bgc compsets. Sample case directories are here:

• /glade/derecho/scratch/wwieder/PLUMBER2_tests/ZM-Mon_spin_SP_test2
• */ZM-Mon_hist_SP_test2
• */ZM-Mon_spin_BGC_test2
• */ZM-Mon_hist_BGC_test2

One outstanding issue may be the format of lai_streams (64-bit offset) and datm inputs (netCDF-4), which are noted int the task-list, above. Currently the model runs with these input data.

The plumber2_usermods script still has some TODO notes that can be addressed, but depending on @TeaganKing's needs we may want to just address this later?

Is this a PR that can come on b4b-dev, or should it come directly to master?

[ekluzek]

64-bit offset is fine. And if the datm NetCDF-4 files are working there isn't an immediate problem. When we try to rimport them, we will need to change the format. But, that can be a simple nccopy exercise done later. And the main problem with NetCDF-4 is likely getting it to work on other machines (I'm not sure it will work on izumi for example).

[ekluzek]

But, yes this should go to b4b-dev. This is an example where simple b4b work can still happen while we are in a holding pattern on master. So another example of a good outcome of adding the b4b-dev branch.

[slevis-lmwg]

My review was addressed, so I'm approving.

[wwieder]

I'll convert some of my TODOS and questions to issues that are more descriptive so we can address them moving forward.

Thanks for helping with this, @ekluzek merging to b4b-dev, @slevis-lmwg

[slevis-lmwg]

1. In @ekluzek's TODO list above, I changed
   "update input data to cdf5 (may currently be netcdf4)" to WON'T FIX FOR NOW

2. python unit tests trigger the following error in 38 of 194 tests. An example looks like this:

ERROR: test_modify_surfdata_atpoint_nocrop_wetland_include_nonveg (test.test_unit_singlept_data_surfdata.TestSinglePointCaseSurfaceNoCrop)
Test modify_surfdata_atpoint for crop cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/work/slevis/git/latest_master/python/ctsm/test/test_unit_singlept_data_surfdata.py", line 519, in test_modify_surfdata_atpoint_nocrop_wetland_include_nonveg
    overwrite=self.overwrite,
TypeError: __init__() missing 2 required positional arguments: 'cth' and 'cbh'

3. python system tests trigger one error:
   ERROR: test_one_site (test.test_sys_run_neon.TestSysRunNeon)
   that says
   ERROR: BUILD FAIL: clm.buildlib failed, cat /glade/derecho/scratch/slevis/tmp/tmp7gcha5xt/BART/bld/lnd.bldlog.240416-093729

4. I also submitted ./run_sys_tests -s aux_clm -c ctsm5.1.dev175 --skip-generate and I see many SHAREDLIB_BUILD errors.

Let me know if you want me to debug (2) (3) and (4).