Add support for QBOi variables #179

chengzhuzhang · 2023-01-05T23:50:03Z

This PR is to support variables requested by QBOi project, using the customized qboi cmor tables. https://gitlab.com/JamesAnstey/qboi-cmor-tables

chengzhuzhang · 2023-01-05T23:51:48Z

@jinboxie here is a development branch for supporting QBOi variables. And i started a template for one variable. I will check if you have permission for this repo and push codes.

codecov · 2023-01-05T23:55:07Z

Codecov Report

Base: 28.98% // Head: 28.93% // Decreases project coverage by -0.05% ⚠️

Coverage data is based on head (2d196b9) compared to base (431638f).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #179      +/-   ##
==========================================
- Coverage   28.98%   28.93%   -0.06%     
==========================================
  Files          93       93              
  Lines        3981     3988       +7     
==========================================
  Hits         1154     1154              
- Misses       2827     2834       +7

Impacted Files	Coverage Δ
e3sm_to_cmip/lib.py	`5.35% <0.00%> (-0.12%)`	⬇️
e3sm_to_cmip/util.py	`32.82% <ø> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

chengzhuzhang · 2023-01-06T20:48:13Z

@jinboxie I updated the utendogw handler. But ran into the issue in cmor:
! Error: You defined variable 'utendogw' (table monZ) with axis id 'longitude' which is not part of this variable according to your table, it says: ( time qboi30 latitude )
It sounds like qboi_cmor table requires only zonal mean of this value?

I ran the code on Cori, with

e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/data -o ~/tests  -v qboi_utendogw -t /global/homes/c/chengzhu/qboi-cmor-tables/Tables/ -u /global/homes/c/chengzhu/e3sm_to_cmip/e3sm_to_cmip/resources/qboi_template.json

jinboxie · 2023-01-06T21:55:43Z

Hi Jill. Yes, the data wanted zonal mean instead of 3D. I reprocessed the data now, please try again.

chengzhuzhang · 2023-01-06T22:38:17Z

I made some updates, now it complains about missing lat_bnds. I think somehow latitude bounds are missing from zonal averaging in your processing.

jinboxie · 2023-01-06T22:47:59Z

@chengzhuzhang Hi Jill. I created a new file during this postprocess zonal mean calculation. It is now added the new bnds and every variables from the previous files. It should be good now.

chengzhuzhang · 2023-01-06T23:42:23Z

Hi @jinboxie it looks like the longitude related data are still present in the data files. I think we picked a hard variable to start, e3sm_to_cmip currently doesn't support zonal mean variables, which is not standard CMIP requests. And implementing it elegantly will take sometime. Another difficulty I have is to actually understand how cmor interact with those QBOi tables, it seems to me cmor is always try to pick up CMIP6 tables, I haven't figured out how to force cmor to pick up other tables.

I have to table this after AMS. At the mean time, I think you can generate a mapping of the conversion, In any case that would be useful either we use e3sm_to_cmip or directly use cmor.

jinboxie · 2023-01-07T20:30:43Z

@chengzhuzhang ok!

chengzhuzhang · 2023-01-25T23:57:37Z

Hi @jinboxie you could look here https://github.com/E3SM-Project/e3sm_to_cmip/pull/179/files for code change I made to temporarily make e3sm_to_cmip to potentially work for the QBOi variable.

You could follow below commands to install e3sm_to_cmip development environment at NERSC,

git clone https://github.com/E3SM-Project/e3sm_to_cmip.git
cd e3sm_to_cmip
conda env create -f conda/dev.yml
conda activate e3sm_to_cmip_dev

get this qboi_support branch and install with python setup.py install.

and following is my command to run on the utendogw variable.

e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/data -o ~/tests -v qboi_utendogw -t /global/homes/c/chengzhu/qboi-cmor-tables/Tables/ -u /global/homes/c/chengzhu/e3sm_to_cmip/e3sm_to_cmip/resources/qboi_template.json

You could try patch the lib.py script and fit the qboi variables..

jinboxie · 2023-01-26T23:36:13Z

Hi @jinboxie you could look here https://github.com/E3SM-Project/e3sm_to_cmip/pull/179/files for code change I made to temporarily make e3sm_to_cmip to potentially work for the QBOi variable.

You could follow below commands to install e3sm_to_cmip development environment at NERSC,
git clone https://github.com/E3SM-Project/e3sm_to_cmip.git
cd e3sm_to_cmip
conda env create -f conda/dev.yml
conda activate e3sm_to_cmip_dev
get this qboi_support branch and install with python setup.py install.

and following is my command to run on the utendogw variable.

e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/data -o ~/tests -v qboi_utendogw -t /global/homes/c/chengzhu/qboi-cmor-tables/Tables/ -u /global/homes/c/chengzhu/e3sm_to_cmip/e3sm_to_cmip/resources/qboi_template.json

You could try patch the lib.py script and fit the qboi variables..

Thank you Jill.

chengzhuzhang · 2023-01-27T00:57:20Z

@jinboxie no problem. I think the lib.py script is not easy to read and perhaps needs some refactoring. I'm also tagging @tomvothecoder here and we can probably think about a re-write in the long run.

jinboxie · 2023-02-20T02:20:32Z

Hi @jinboxie you could look here https://github.com/E3SM-Project/e3sm_to_cmip/pull/179/files for code change I made to temporarily make e3sm_to_cmip to potentially work for the QBOi variable.
You could follow below commands to install e3sm_to_cmip development environment at NERSC,
git clone https://github.com/E3SM-Project/e3sm_to_cmip.git
cd e3sm_to_cmip
conda env create -f conda/dev.yml
conda activate e3sm_to_cmip_dev
get this qboi_support branch and install with python setup.py install.
and following is my command to run on the utendogw variable.
e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/data -o ~/tests -v qboi_utendogw -t /global/homes/c/chengzhu/qboi-cmor-tables/Tables/ -u /global/homes/c/chengzhu/e3sm_to_cmip/e3sm_to_cmip/resources/qboi_template.json
You could try patch the lib.py script and fit the qboi variables..
Thank you Jill.

Hi Jill @chengzhuzhang . I'm having a problem with the command.
e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/data -o ~/tests -v qboi_utendogw -t /global/cscratch1/sd/xie7/forJill/qboi-cmor-tables-master/Tables/ -u /global/cscratch1/sd/xie7/qboi/TEM_calculate/qboi-cmip6/e3sm_to_cmip/e3sm_to_cmip/resources/qboi_template.json

this was even the same with ua for some reason
e3sm_to_cmip -i /global/cscratch1/sd/xie7/forJill/ -o /global/cscratch1/sd/xie7/qboi/TEM_calculate/qboi-cmip6/e3sm_to_cmip/tests -v ua -t /global/cscratch1/sd/xie7/qboi/TEM_calculate/qboi-cmip6/e3sm_to_cmip/e3sm_to_cmip/resources -u /global/cscratch1/sd/xie7/qboi/TEM_calculate/qboi-cmip6/e3sm_to_cmip/e3sm_to_cmip/resources/CMIP6_Amon.json
, do you know what's wrong or where I can check with?

0%| | 0/1 [00:00<?, ?it/s]
*** buffer overflow detected ***: /global/cscratch1/sd/xie7/apps/Miniconda3/envs/e3sm_to_cmip_dev/
bin/python terminated
File "/global/cscratch1/sd/xie7/apps/Miniconda3/envs/e3sm_to_cmip_dev/lib/python3.10/site-packag
es/e3sm_to_cmip-1.9.1-py3.10.egg/e3sm_to_cmip/lib.py", line 106, in run_parallel
out = res.result() File "/global/cscratch1/sd/xie7/apps/Miniconda3/envs/e3sm_to_cmip_dev/lib/python3.10/concurrent/
futures/base.py", line 458, in result return self.get_result()
File "/global/cscratch1/sd/xie7/apps/Miniconda3/envs/e3sm_to_cmip_dev/lib/python3.10/concurrent/
futures/base.py", line 403, in get_result
raise self.exception A process in the process pool was terminated abruptly while the future was running or pending.
100%|________________________________________________________| 1/1 [00:00<00:00, 4.04it/s][+] 0 of 1 handlers complete
[-] ua failed to complete

chengzhuzhang · 2023-02-21T18:13:58Z

@jinboxie , based on our discussion. It is best to start from standard e3sm_to_cmip use case, so please download the CMIP tables from here: https://github.com/PCMDI/cmip6-cmor-tables and point to downloaded tables by using -t. This branch has some changes specifically to accommodate qboi variables. You can switch to use e3sm_to_cmip master to test the standard ua variable. Let me know if there are any problems.

jinboxie · 2023-03-08T21:47:13Z

Hi Jill @chengzhuzhang , please refer to https://github.com/E3SM-Project/e3sm_to_cmip/tree/jinboxie_qboi
for the current version that's successful in implementing QBOi json files.

chengzhuzhang · 2023-03-08T22:03:27Z

Thank you, @jinboxie !

template for qboi variables

bb435e4

update utendogw

c5c03cd

temporal patch for aaccomodating( time plev lat) dims

2d196b9

tomvothecoder added this to the FY25 Development milestone Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for QBOi variables #179

Add support for QBOi variables #179

chengzhuzhang commented Jan 5, 2023

chengzhuzhang commented Jan 5, 2023 •

edited

Loading

codecov bot commented Jan 5, 2023 •

edited

Loading

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 6, 2023

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 6, 2023

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 7, 2023

chengzhuzhang commented Jan 25, 2023 •

edited

Loading

jinboxie commented Jan 26, 2023

chengzhuzhang commented Jan 27, 2023

jinboxie commented Feb 20, 2023 •

edited

Loading

chengzhuzhang commented Feb 21, 2023

jinboxie commented Mar 8, 2023 •

edited

Loading

chengzhuzhang commented Mar 8, 2023

Add support for QBOi variables #179

Are you sure you want to change the base?

Add support for QBOi variables #179

Conversation

chengzhuzhang commented Jan 5, 2023

chengzhuzhang commented Jan 5, 2023 • edited Loading

codecov bot commented Jan 5, 2023 • edited Loading

Codecov Report

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 6, 2023

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 6, 2023

chengzhuzhang commented Jan 6, 2023

jinboxie commented Jan 7, 2023

chengzhuzhang commented Jan 25, 2023 • edited Loading

jinboxie commented Jan 26, 2023

chengzhuzhang commented Jan 27, 2023

jinboxie commented Feb 20, 2023 • edited Loading

chengzhuzhang commented Feb 21, 2023

jinboxie commented Mar 8, 2023 • edited Loading

chengzhuzhang commented Mar 8, 2023

chengzhuzhang commented Jan 5, 2023 •

edited

Loading

codecov bot commented Jan 5, 2023 •

edited

Loading

chengzhuzhang commented Jan 25, 2023 •

edited

Loading

jinboxie commented Feb 20, 2023 •

edited

Loading

jinboxie commented Mar 8, 2023 •

edited

Loading