ILAMB task #197

chengzhuzhang · 2022-02-17T18:26:51Z

This PR is to add capability to run ILAMB package for land model bench-marking. Close #133, #134.

Example configuration file:

[default]
input = /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis
input_subdir = archive/atm/hist
output = /lcrc/group/e3sm/ac.zhang40/zppy_tests/zppy_ilamb_324_tr1 
case = 20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/zppy_ilamb
partition = compute
# TODO: will add a configuration template for "BGC" campaign
campaign = "water_cycle"
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"

# Regridded monthly time series files in 10 year chunks
[ts]
active = True
years = "1:10:10",
ts_fmt = "cmip"

  # Monthly land time series
  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # TODO: add more variables requested by BGC
  vars = "LAISHA,LAISUN"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # TODO: add more variables requested by BGC
  vars = "PRECC,PRECL"

[ilamb_run]
active = True
years = "1:10:10",
ts_num_years = 10
short_name = '20210122.v2_test01'
  [[ land_monthly ]]

Integrate in e3sm_to_cmip function for additional parameter to generate CMIP-like time-series for land and atmosphere variables, that ilamb_run task required.
Integrate in ilamb_run task.
TODO: Include equivalent mpi call for supported machines.
TODO: Copy files to web servers.
TODO: figure out path for ILAMB obs data.

External development for e3sm_to_cmip

TODO: Add more variable conversion handlers. Current available handlers in e3sm_to_cmip
'mrsos, mrso, mrfso, mrros, mrro, prveg, evspsblveg, evspsblsoi, tran, tsl, lai, cLitter, cProduct, cSoilFast,cSoilMedium,cSoilSlow fFire, fHarvest, cVeg, nbp, gpp, ra, rh'
Conversion formula should be added here: https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/925500501/Lmon+variable+conversion+table

ILAMB has some underlying assumptions and are noted here (Thanks to @minxu74, one of the ILAMB developers, to help me understand):

The benchmarking practice is recommended applying to historical simulations, i.e. simulation data at least partially over the time period of the source data (year 2000-). ILAMB will match the overlap of the model and obs years, except using "--model_year" to shift year to create overlapping. ILAMB can work with continuous time segments without pre-concantenation.
if there is no areacella, ILAMB will use longitude and latitude information to compute areas (CellAreas function in ilamblib.py); if the sftlf variable is missing, 100% land area is assumed. This is an approximate method.
the cmip.cfg will be used as base configuration file. This .cfg is specific for CMIP5 and 6 benchmarking. The other file ilamb.cfg is used to run ilamb in a default setting, some metrics are removed in the cmip.cfg as the observational data’s uncertainty is large from feedbacks of users and rubisco team members. But I will defer to BGC to provide a more specific ilamb config to fit the research need.

chengzhuzhang · 2022-03-17T22:23:26Z

Update Mar 17, 2022:
A prototype is working on Chrysalis. More TODO

Need a short name option to avoid using long case name as model name. example: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.zhang40/zppy_ilamb/tests_0317/
Make the simple mode from e3sm_to_cmip work, which allows only convert variables without attaching CMIP metadata, otherwise I have to include templates of cmor tables and meta-data files

chengzhuzhang · 2022-03-25T16:43:56Z

Hey @forsyth2 , this PR is ready for review. The example cfg to test a run can be found here:#197 (comment)
There is a unit test error from CI/CD, but i'm not exactly sure how to fix.
And as we discussed, this PR only covers running Chrysalis, would you please extend this PR or have an enhancement PR to address those machine specific items: path of obs, copy to web servers, and MPI calls...Thank you!

forsyth2 · 2022-03-25T18:46:11Z

@chengzhuzhang Thanks! I will review and test. I also created #229 to support the other machines.

forsyth2

From what I can tell, the code looks mostly alright -- I just made a few comments.

I will test your configuration file and run the unit/integration tests on Chrysalis once it's back online.

forsyth2 · 2022-03-25T21:32:54Z

zppy/ilamb_run.py

+                os.path.join(
+                    scriptDir,
+                    "ts_%s_%04d-%04d-%04d.status"
+                    % ("land_monthly", c["year1"], c["year2"], c["ts_num_years"]),


Is "land_monthly" always going to be the subsection?

I meant to have "land_monthly" the minimum dependency. One can have "atm_monthly" data processed, but not required.

forsyth2 · 2022-03-25T21:33:31Z

zppy/ilamb_run.py

+                        scriptDir,
+                        "ts_%s_%04d-%04d-%04d.status"
+                        % (
+                            "atm_monthly_180x360_aave",


Is "atm_monthly_180x360_aave" always going to be the subsection?

no, same comment as above..

forsyth2 · 2022-03-25T21:36:34Z

zppy/templates/ilamb_run.bash

+fi
+
+# Point to obsersvation data
+# TODO: need to update these data to other supported machines


Note: Will be done in #229

forsyth2 · 2022-03-25T21:38:41Z

zppy/templates/ilamb_run.bash

+echo ${workdir}
+echo {{ scriptDir }}
+
+srun --export=ALL -N 1 ilamb-run --config ilamb_run.cfg --model_root $model_root  --regions global --model_year ${Y1} 2000


I think you could just put export="ALL" in your submitScript call rather than running srun here.

Yes, it makes sense! maybe this can be handled in #229 as well?

forsyth2 · 2022-03-25T21:39:20Z

zppy/templates/ilamb_run.bash

+
+# Run diagnostics
+# Note --export=All is needed to make sure the executable is copied and executed  on the nodes.
+# TODO: find the mpi run format for different platforms


Note: Will be done in #229.

forsyth2 · 2022-03-25T21:40:05Z

zppy/templates/ilamb_run.bash

+  echo 'ERROR (2)' > {{ prefix }}.status
+  exit 2


Since this is the first exit, let's do ERROR (1) and exit 1.

forsyth2 · 2022-03-25T21:40:51Z

zppy/templates/ilamb_run.bash

+echo ===== COPY FILES TO WEB SERVER =====
+echo
+
+# TODO copy _build from $workdir to web server


Is the output copied to the web server in another piece of code somewhere else?

No. The output is still in workdir.

forsyth2 · 2022-03-28T19:33:41Z

Now that Chrysalis is back up, I ran your configuration file and the unit/integration tests:

This is my version of your configuration file:

[default]
campaign = "water_cycle"
case = 20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
input = /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis
input_subdir = archive/atm/hist
output = /lcrc/group/e3sm/ac.forsyth2/zppy_development/pr_197.example
partition = compute
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_development/pr_197

# Regridded monthly time series files in 10 year chunks
[ts]
active = True
mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
ts_fmt = "cmip"
years = "1:10:10",

  # Monthly land time series
  [[ land_monthly ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "LAISHA,LAISUN"

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  vars = "PRECC,PRECL"

[ilamb_run]
active = True
short_name = '20210122.v2_test01'
ts_num_years = 10
years = "1:10:10",

  [[ land_monthly ]]

That causes Errors occurred in the run, please consult ./_build/ILAMB01.log for more detailed information.

The unit tests need to be updated as do the expected files for the integration tests. test_complete_run has an error rather than a failure:
FileNotFoundError: [Errno 2] No such file or directory: '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_complete_run_www/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1850-1853/viewer/viewer/e3sm_logo.png'

forsyth2 · 2022-03-28T19:34:28Z

@chengzhuzhang Since this branch is off the main repo rather than a fork, I think it should be easy for me to push further changes to it. Would you like me to make the fixes I suggested?

chengzhuzhang · 2022-03-28T19:42:46Z

@chengzhuzhang Since this branch is off the main repo rather than a fork, I think it should be easy for me to push further changes to it. Would you like me to make the fixes I suggested?

Yes, please! I'm also not familiar with zppy's tests, please go ahead and make the fixes for me. Thank you!

forsyth2 · 2022-03-28T23:15:05Z

@chengzhuzhang Do you know why I got Errors occurred in the run, please consult ./_build/ILAMB01.log for more detailed information? Is there something I need to change in the config file? I see a lot of ILAMB.ilamblib.VarNotInModel: VarNotInModel in that log file.

chengzhuzhang · 2022-03-28T23:21:30Z

Those are ilamb error logs tell that some expected variables are not located. The tested configuration only included a minimal sets variables. So 'ILAMB.ilamblib.VarNotInModel: VarNotInModel' is expected. did your run actually create a index.html with some results?

forsyth2 · 2022-03-28T23:29:36Z

did your run actually create a index.html with some results?

Ah ok. I moved my results to https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_development/pr_197/_build/. Most of it is missing data except for some leaf-area-index values, which matches the variables given, so I think that works then.

forsyth2 · 2022-04-06T23:03:32Z

Closed with #230.

chengzhuzhang added 2 commits February 15, 2022 18:10

integrate CMIP time-series generation

39f73f7

ilamb_run_prototype

b2cc73e

forsyth2 added the semver: new feature New feature (will increment minor version) label Mar 3, 2022

add ilamb configuration files; running on chrysalis

3e86d10

chengzhuzhang added 5 commits March 17, 2022 17:51

update obs data path

8b9e57d

resolve confict

3493e6d

fix prc_typ formatting

2425aa6

attach CMIP metadata

ded1f9f

add cmip metadata file

ed8ecf9

chengzhuzhang marked this pull request as ready for review March 25, 2022 16:37

chengzhuzhang requested a review from forsyth2 March 25, 2022 16:39

forsyth2 mentioned this pull request Mar 25, 2022

Support ILAMB on other machines #229

Closed

forsyth2 reviewed Mar 25, 2022

View reviewed changes

forsyth2 mentioned this pull request Mar 28, 2022

ILAMB task revision #230

Merged

chengzhuzhang changed the title ~~Adding ilamb_run task~~ ILAMB task Mar 29, 2022

forsyth2 closed this Apr 6, 2022

forsyth2 deleted the e3sm2cmip_ilamb branch May 11, 2022 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ILAMB task #197

ILAMB task #197

chengzhuzhang commented Feb 17, 2022 •

edited

Loading

chengzhuzhang commented Mar 17, 2022 •

edited

Loading

chengzhuzhang commented Mar 25, 2022

forsyth2 commented Mar 25, 2022

forsyth2 left a comment

forsyth2 Mar 25, 2022

chengzhuzhang Mar 28, 2022

forsyth2 Mar 25, 2022

chengzhuzhang Mar 28, 2022

forsyth2 Mar 25, 2022

forsyth2 Mar 25, 2022

chengzhuzhang Mar 28, 2022

forsyth2 Mar 25, 2022

forsyth2 Mar 25, 2022

forsyth2 Mar 25, 2022

chengzhuzhang Mar 28, 2022

forsyth2 commented Mar 28, 2022

forsyth2 commented Mar 28, 2022

chengzhuzhang commented Mar 28, 2022

forsyth2 commented Mar 28, 2022

chengzhuzhang commented Mar 28, 2022 •

edited

Loading

forsyth2 commented Mar 28, 2022

forsyth2 commented Apr 6, 2022

ILAMB task #197

ILAMB task #197

Conversation

chengzhuzhang commented Feb 17, 2022 • edited Loading

chengzhuzhang commented Mar 17, 2022 • edited Loading

chengzhuzhang commented Mar 25, 2022

forsyth2 commented Mar 25, 2022

forsyth2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

forsyth2 commented Mar 28, 2022

forsyth2 commented Mar 28, 2022

chengzhuzhang commented Mar 28, 2022

forsyth2 commented Mar 28, 2022

chengzhuzhang commented Mar 28, 2022 • edited Loading

forsyth2 commented Mar 28, 2022

forsyth2 commented Apr 6, 2022

chengzhuzhang commented Feb 17, 2022 •

edited

Loading

chengzhuzhang commented Mar 17, 2022 •

edited

Loading

chengzhuzhang commented Mar 28, 2022 •

edited

Loading