Add example script #174

chengzhuzhang · 2022-12-02T21:30:56Z

Provide an end-to-end example script that is independent from CWL workflows.
This script serves for a few purpose:

establish the data and script required by integration tests.
provide necessary commands for interfacing with zppy, with the hope that zppy can create and archive intermediate files required as e3sm_to_cmip input.

TO DO:

construct unified data directory in coordinate with @tomvothecoder 's refactoring effort. 9b5ed23
document input data requirement for running e3sm_to_cmip. e331b55

codecov · 2022-12-02T21:34:53Z

Codecov Report

Base: 28.98% // Head: 28.98% // No change to project coverage 👍

Coverage data is based on head (04ee5b0) compared to base (e95a74b).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #174   +/-   ##
=======================================
  Coverage   28.98%   28.98%           
=======================================
  Files          93       93           
  Lines        3981     3981           
=======================================
  Hits         1154     1154           
  Misses       2827     2827

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

tomvothecoder · 2022-12-05T16:45:27Z

This is much needed because we should test changes/debug directly through e3sm_to_cmip rather than the CWL workflows.

chengzhuzhang · 2022-12-20T02:03:10Z

scripts/example_end-to-end_script.sh

+e3sm_to_cmip -i ${rgr_dir} -o $result_dir -v ${cmip_var_list} -t --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables -u ${metadata_path}
+
+# CMORIZE Sea-ice Monthly variables 
+# Note the input folder for mpas sea ice files requires:


@xylar I started to understand the e2c better. would you help review if the following is accurate? Could you let me know the Constants that are required from mpaso_in.

At present, the sea-ice handlers never use constants from the mpassi_in namelist (note: they would not use mpaso_in as you suggested above). Presumably, the option was implemented in case they ever would need these constants as MPAS-Ocean does.

chengzhuzhang · 2022-12-20T02:04:12Z

scripts/example_end-to-end_script.sh

+e3sm_to_cmip -s --realm SImon --var-list siconc, sitemptop, sisnmass, sitimefrac, siu, siv, sithick, sisnthick, simass --map ${e2c_path}/maps/map_EC30to60E2r2_to_cmip6_180x360_aave.20220301.nc --input-path ${model_data}/v2.mpassi_input/ --output-path ${result_dir} --user-metadata ${metadata_path} --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables
+
+
+# CMORIZE Ocean Monthly variables 


@xylar Same here, would you also look at this session for MPAS ocean processing input requirement?

The only constant needed from mpaso_in is config_density0:

$ cd e3sm_to_cmip/cmor_handlers $ grep -r "namelist\[" hfsifrazil.py: config_density0 = float(namelist['config_density0']) hfsifrazil.py: float(namelist['config_frazil_heat_of_fusion']) msftmz.py: config_density0 = float(namelist['config_density0']) pbo.py: config_density0 = float(namelist['config_density0']) masso.py: config_density0 = float(namelist['config_density0']) pso.py: config_density0 = float(namelist['config_density0']) masscello.py: config_density0 = float(namelist['config_density0'])

This is the reference density of the ocean, used to convert from volume to mass.

it looks like hfsifrazil requires both 'config_density0' and 'config_frazil_heat_of_fusion'.

chengzhuzhang · 2022-12-20T02:12:28Z

@tomvothecoder Hi I think this PR should be ready to go after we get some additional input from Xylar. I think we can either setting up the integration on acme1 or lcrc.

tomvothecoder · 2022-12-21T17:31:53Z

Sounds good @chengzhuzhang. Thanks!

chengzhuzhang · 2022-12-21T17:47:56Z

@xylar Hey Xylar, I updated the comment to provide needed constants from the mpaso namelist file. Please help review and let me know if anything that I missed...Thank you.

xylar

Looks good. A few small suggestions and answers to your questions.

scripts/example_end-to-end_script.sh

xylar · 2022-12-23T11:00:37Z

scripts/example_end-to-end_script.sh

+# 4. A region masks file for MOC regions: EC30to60E2r2_mocBasinsAndTransects20210623.nc (Needed for variable msftmz: Ocean Meridional Overturning Mass Streamfunction)
+e3sm_to_cmip -s --realm Omon --var-list areacello, fsitherm, hfds, masso, mlotst, sfdsi, sob, soga, sos, sosga, tauuo, tauvo, thetaoga, tob, tos, tosga, volo, wfo, zos, thetaoga, hfsifrazil, masscello, so, thetao, thkcello, uo, vo, volcello, wo zhalfo --map ${e2c_path}/maps/map_EC30to60E2r2_to_cmip6_180x360_aave.20220301.nc --input-path ${model_data}/v2.mpaso_input/ --output-path ${result_dir} --user-metadata ${metadata_path} --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables
+
+exit


(not needed)

xylar · 2022-12-23T11:08:01Z

scripts/example_end-to-end_script.sh

+e3sm_to_cmip -i ${rgr_dir} -o $result_dir -v ${cmip_var_list} -t --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables -u ${metadata_path}
+
+# CMORIZE Sea-ice Monthly variables 
+# Note the input folder for mpas sea ice files requires:


At present, the sea-ice handlers never use constants from the mpassi_in namelist (note: they would not use mpaso_in as you suggested above). Presumably, the option was implemented in case they ever would need these constants as MPAS-Ocean does.

xylar · 2022-12-23T11:09:38Z

scripts/example_end-to-end_script.sh

+e3sm_to_cmip -s --realm SImon --var-list siconc, sitemptop, sisnmass, sitimefrac, siu, siv, sithick, sisnthick, simass --map ${e2c_path}/maps/map_EC30to60E2r2_to_cmip6_180x360_aave.20220301.nc --input-path ${model_data}/v2.mpassi_input/ --output-path ${result_dir} --user-metadata ${metadata_path} --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables
+
+
+# CMORIZE Ocean Monthly variables 


The only constant needed from mpaso_in is config_density0:

$ cd e3sm_to_cmip/cmor_handlers $ grep -r "namelist\[" hfsifrazil.py: config_density0 = float(namelist['config_density0']) hfsifrazil.py: float(namelist['config_frazil_heat_of_fusion']) msftmz.py: config_density0 = float(namelist['config_density0']) pbo.py: config_density0 = float(namelist['config_density0']) masso.py: config_density0 = float(namelist['config_density0']) pso.py: config_density0 = float(namelist['config_density0']) masscello.py: config_density0 = float(namelist['config_density0'])

This is the reference density of the ocean, used to convert from volume to mass.

Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

tomvothecoder · 2023-01-30T19:13:53Z

Hey @chengzhuzhang, I just got back to working on #103 and am leveraging this script to generate CMORized data.

I am currently experimenting with the different blocks of code in the example script to generate each set of CMORized data.
In the meantime, can you provide a step-by-step on how to use this script to streamline the process? It would be useful to have in our documentation for future reference. Thanks!

chengzhuzhang · 2023-01-30T23:34:19Z

Hi @tomvothecoder thanks for resuming work #103, and let me know if there is any difficulty in testing the example script.
I agree it will be helpful to have a step-by-step guide. The limitation of this script right now is that it is designed for testing purpose, it works on a set of prepared directories with required data already staged. And in a real use-case, an extra step is needed to move/create symlinks for needed data with specified time period. I will try to generate that and include in our documentation.

add example script

fa55a2c

clean up

c8dbfec

chengzhuzhang added 3 commits December 19, 2022 16:04

consolidate paths

9b5ed23

fix more paths

36eff73

add some docs for input files

e331b55

chengzhuzhang marked this pull request as ready for review December 20, 2022 02:00

chengzhuzhang requested review from xylar and tomvothecoder December 20, 2022 02:00

chengzhuzhang commented Dec 20, 2022

View reviewed changes

update

866303e

xylar approved these changes Dec 23, 2022

View reviewed changes

chengzhuzhang and others added 5 commits January 3, 2023 11:39

Update scripts/example_end-to-end_script.sh

7a6ebb5

Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Update scripts/example_end-to-end_script.sh

3e465ec

Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Update scripts/example_end-to-end_script.sh

4e4136c

Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Update scripts/example_end-to-end_script.sh

89a4c5e

Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Merge branch 'master' into end-to-end_example

04ee5b0

chengzhuzhang merged commit 431638f into master Jan 3, 2023

tomvothecoder mentioned this pull request Jan 26, 2023

Refactor the API design for cmor_handlers #140

Merged

40 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add example script #174

Add example script #174

chengzhuzhang commented Dec 2, 2022 •

edited

Loading

codecov bot commented Dec 2, 2022 •

edited

Loading

tomvothecoder commented Dec 5, 2022

chengzhuzhang Dec 20, 2022

xylar Dec 23, 2022

chengzhuzhang Dec 20, 2022

xylar Dec 23, 2022

chengzhuzhang Jan 3, 2023

chengzhuzhang commented Dec 20, 2022

tomvothecoder commented Dec 21, 2022

chengzhuzhang commented Dec 21, 2022

xylar left a comment

xylar Dec 23, 2022

xylar Dec 23, 2022

xylar Dec 23, 2022

tomvothecoder commented Jan 30, 2023 •

edited

Loading

chengzhuzhang commented Jan 30, 2023

		e3sm_to_cmip -s --realm SImon --var-list siconc, sitemptop, sisnmass, sitimefrac, siu, siv, sithick, sisnthick, simass --map ${e2c_path}/maps/map_EC30to60E2r2_to_cmip6_180x360_aave.20220301.nc --input-path ${model_data}/v2.mpassi_input/ --output-path ${result_dir} --user-metadata ${metadata_path} --tables-path ${e2c_path}/cmor/cmip6-cmor-tables/Tables


		# CMORIZE Ocean Monthly variables

Add example script #174

Add example script #174

Conversation

chengzhuzhang commented Dec 2, 2022 • edited Loading

codecov bot commented Dec 2, 2022 • edited Loading

Codecov Report

tomvothecoder commented Dec 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chengzhuzhang commented Dec 20, 2022

tomvothecoder commented Dec 21, 2022

chengzhuzhang commented Dec 21, 2022

xylar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomvothecoder commented Jan 30, 2023 • edited Loading

chengzhuzhang commented Jan 30, 2023

chengzhuzhang commented Dec 2, 2022 •

edited

Loading

codecov bot commented Dec 2, 2022 •

edited

Loading

tomvothecoder commented Jan 30, 2023 •

edited

Loading