CMORizer for JRA-55 #3141

axel-lauer · 2023-03-28T12:15:20Z

Description

This PR adds downloading and formatting scripts for the JRA-55 reanalysis. JRA-55 data have been provided long ago as ana4mips dataset that is no longer available (or maintained). In order to add new variables, the original JRA-55 data have to be CMORized. The original data are only available as GRIB1.

Note: This PR requires the Python library cfgrib to process the GRIB files, e.g. conda install cfgrib

This PR allows processing of the following JRA-55 variables:

cli
clivi
clw
clwvi
clt
prw
rlus
rlut
rlutcs
rsus
rsuscs
rsut
rsutcs
tas
wap

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 Any changed dependencies have been added or removed correctly
🛠 All checks below this pull request were successful

New or updated data reformatting script

🛠 Documentation is available
🛠 The dataset has been added to the CMOR check recipe
🧪 Numbers and units of the data look physically meaningful

alistairsellar · 2023-07-20T10:40:54Z

Hi @axel-lauer are you still looking for a science reviewer for this one? I could do this...

hb326 · 2023-07-20T11:30:03Z

Hi @axel-lauer are you still looking for a science reviewer for this one? I could do this...

@alistairsellar: Axel is on vacation right now, but I know that he is still looking for a science reviewer for this. So thanks for offering! :)

remi-kazeroni

Nice work, @axel-lauer! The code looks good to me and all works fine: downloader, formatter, recipe_check_obs. I only have one suggestion for a codacy issue, the others could be ignored. The downloading script follows recommendations on the source server and worked fine for me.

I would have given a technical approval if the tests were still passing but that is no longer the case after the main branch was last merged into this one. There seems to be an issue with the iris-grib package. It could be an issue with an upstream package rather than how the package is used in this CMORizer. I will try to take a deeper look next week and report here what I could figure out.

esmvaltool/cmorizers/data/formatters/datasets/jra_55.py

remi-kazeroni · 2023-08-21T15:28:56Z

I could get the dependency tests pass locally by adding python-eccodes<1.6.0 to environment*yml and setup.py which is needed to install iris-grid. The tests would still fail locally with python-eccodes not pinned (i.e. 1.6.0).

For the latest version of python-eccodes, I still get ImportError: /opt/conda/envs/esmvaltool/lib/python3.11/site-packages/gribapi/_bindings.cpython-311-x86_64-linux-gnu.so: undefined symbol: codes_bufr_key_is_coordinate which seems to be an issue upstream.

If I understand correctly, the python-eccodes would need to be rebuilt on conda-forge to be compatible with the latest eccodes (2.30.2, but still at 2.21.2 on conda-forge). But perhaps our in-house conda-forge specialists, @zklaus and @valeriupredoi, would know better how to best handle eccodes and the Python bindings for the Tool 🍻

alistairsellar

Hi @axel-lauer looks good for science review. Or at least I'm happy with code (pending response to suggestions) & doc changes (can you link documentation to PR description).

Should science review cover some looking at output files? Presumably I would need to make a new environment to run this, as standard esmvaltool modules (e.g. on Jasmin) won't yet have iris-grib?

esmvaltool/cmorizers/data/formatters/datasets/jra_55.py

esmvaltool/cmorizers/data/cmor_config/JRA-55.yml

esmvaltool/cmorizers/data/formatters/datasets/jra_55.py

alistairsellar · 2023-08-30T14:51:36Z

esmvaltool/cmorizers/data/formatters/datasets/jra_55.py

+    utils.fix_var_metadata(cube, cmor_info)
+
+    # fix z-coordinate (if present)
+


Redundant blank line?

Suggested change

Thanks for the other changes @axel-lauer. Not sure if this one got missed or you want to keep the blank line for a reason? It would now be line 135 that this refers to.

zklaus · 2023-09-01T09:50:36Z

I could get the dependency tests pass locally by adding python-eccodes<1.6.0 to environment*yml and setup.py which is needed to install iris-grid. The tests would still fail locally with python-eccodes not pinned (i.e. 1.6.0).

For the latest version of python-eccodes, I still get ImportError: /opt/conda/envs/esmvaltool/lib/python3.11/site-packages/gribapi/_bindings.cpython-311-x86_64-linux-gnu.so: undefined symbol: codes_bufr_key_is_coordinate which seems to be an issue upstream.

If I understand correctly, the python-eccodes would need to be rebuilt on conda-forge to be compatible with the latest eccodes (2.30.2, but still at 2.21.2 on conda-forge). But perhaps our in-house conda-forge specialists, @zklaus and @valeriupredoi, would know better how to best handle eccodes and the Python bindings for the Tool 🍻

Thanks for bringing this up! I dug a bit deeper and it looks like this python-eccodes upstream (at ECMWF) declares a min_recommended_version for Eccodes which currently stands at 2.21.0. However, with release 1.6.0 they added linking to a function that was only added to Eccodes in 2.31.0. Conda-forge relies on the declaration in python-eccodes to do the pinning, so I have opened an issue there (ecmwf/eccodes-python#82) to update that. In the meantime, we can already fix the conda-forge package, which is tracked in conda-forge/python-eccodes-feedstock#86.

Could you possibly test if things work with python-eccodes-1.6.0 and eccodes-2.31.0?

Co-authored-by: Rémi Kazeroni <remi.kazeroni@gmail.com>

alistairsellar

Thanks for making changes @axel-lauer. Apart from one outstanding question about a blank line (see above), I think that covers my review, so marking as approved. Great stuff!

axel-lauer · 2023-09-28T04:43:17Z

Thanks for your review @alistairsellar ! I removed the blank line as suggested.

schlunma · 2023-10-10T13:42:36Z

Could you possibly test if things work with python-eccodes-1.6.0 and eccodes-2.31.0?

@zklaus let's discuss this in ESMValGroup/ESMValCore#2178, it seems this PR is no longer using iris-grib.

axel-lauer · 2023-10-10T14:50:05Z

Could you possibly test if things work with python-eccodes-1.6.0 and eccodes-2.31.0?

@zklaus let's discuss this in ESMValGroup/ESMValCore#2178, it seems this PR is no longer using iris-grib.

This is correct, I replaced iris-grib with xarray/cfgrib, which is able to distinguish between different level types (in contrast to iris-grib). This makes the code much easier to read and maintain.

flicj191

Data downloaded and CMORized on Gadi, thanks

flicj191 · 2024-05-09T06:22:04Z

esmvaltool/cmorizers/data/downloaders/datasets/jra_55.py

+
+    os.makedirs(downloader.local_folder, exist_ok=True)
+
+    user = os.environ.get("rda-user")


Authentication methods may have changed on RDA, my login uses my orcid login, so this user, password wasn't working for me. But I commented out all the downloader.login() (lines 43-56, 64-69) stuff and was able to download fine on Gadi supercomputer.

rbeucher

Had a look at this and it looks good to go

axel-lauer added 8 commits March 22, 2023 07:13

first version

bbf8d5f

first working version of jra-55 downloader

370e516

fixed flake8 issues

3b88dec

snapshot 2023-03-23

2e08769

first complete version

15a848d

fixed flake issues

6b7aa5b

added iris-grib to environment*.yml

eacb84c

added iris-grib to setup.py

e401a51

axel-lauer added the Data label Mar 28, 2023

added variable ta to JRA-55 CMORizer

67f6cf1

axel-lauer added looking for technical reviewer looking for scientific reviewer labels May 4, 2023

axel-lauer requested review from LisaBock, schlunma and hb326 May 4, 2023 12:51

axel-lauer marked this pull request as ready for review May 4, 2023 12:51

axel-lauer requested a review from remi-kazeroni as a code owner May 4, 2023 12:51

axel-lauer added 3 commits June 1, 2023 14:11

Merge branch 'main' into cmorizer_jra-55

a6a8b45

fixed issues in downloader

2818967

merged latest main branch

f603f54

alistairsellar self-requested a review July 20, 2023 16:12

alistairsellar removed the looking for scientific reviewer label Jul 20, 2023

remi-kazeroni added observations in technical review in scientific review and removed looking for technical reviewer labels Aug 14, 2023

axel-lauer and others added 2 commits August 17, 2023 13:51

updated download path (changed by RDA)

c85c45d

Merge branch 'main' into cmorizer_jra-55

0fbd84a

remi-kazeroni reviewed Aug 18, 2023

View reviewed changes

esmvaltool/cmorizers/data/formatters/datasets/jra_55.py Outdated Show resolved Hide resolved

schlunma mentioned this pull request Aug 25, 2023

Add support for native ERA5 data in GRIB format ESMValGroup/ESMValCore#2178

Merged

15 tasks

alistairsellar requested changes Aug 31, 2023

View reviewed changes

axel-lauer and others added 4 commits September 25, 2023 15:01

Update esmvaltool/cmorizers/data/formatters/datasets/jra_55.py

a039383

Co-authored-by: Rémi Kazeroni <remi.kazeroni@gmail.com>

JRA-55 formatter: added support for start/end (year)

8d33497

JRA-55 formatter: simplified code for operator diff

332de23

JRA-55 formatter: switched from iris-grib to xarray

76fff77

axel-lauer requested a review from a team as a code owner September 27, 2023 11:41

alistairsellar approved these changes Sep 27, 2023

View reviewed changes

removed blank line(s)

bb1a307

axel-lauer and others added 2 commits December 13, 2023 09:33

Merge branch 'main' into cmorizer_jra-55

792537d

Merge branch 'main' into cmorizer_jra-55

dd28b2f

rbeucher assigned flicj191 Feb 20, 2024

Merge branch 'main' into cmorizer_jra-55

10b92cd

flicj191 reviewed May 9, 2024

View reviewed changes

rbeucher approved these changes Oct 1, 2024

View reviewed changes

rbeucher merged commit 6155acf into main Oct 1, 2024
8 of 9 checks passed

rbeucher deleted the cmorizer_jra-55 branch October 1, 2024 23:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMORizer for JRA-55 #3141

CMORizer for JRA-55 #3141

axel-lauer commented Mar 28, 2023 •

edited

Loading

alistairsellar commented Jul 20, 2023

hb326 commented Jul 20, 2023

remi-kazeroni left a comment

remi-kazeroni commented Aug 21, 2023

alistairsellar left a comment

alistairsellar Aug 30, 2023

alistairsellar Sep 27, 2023

zklaus commented Sep 1, 2023

alistairsellar left a comment

axel-lauer commented Sep 28, 2023

schlunma commented Oct 10, 2023

axel-lauer commented Oct 10, 2023

flicj191 left a comment

flicj191 May 9, 2024

rbeucher left a comment

		utils.fix_var_metadata(cube, cmor_info)

		# fix z-coordinate (if present)


		os.makedirs(downloader.local_folder, exist_ok=True)

		user = os.environ.get("rda-user")

CMORizer for JRA-55 #3141

CMORizer for JRA-55 #3141

Conversation

axel-lauer commented Mar 28, 2023 • edited Loading

Description

Checklist

New or updated data reformatting script

alistairsellar commented Jul 20, 2023

hb326 commented Jul 20, 2023

remi-kazeroni left a comment

Choose a reason for hiding this comment

remi-kazeroni commented Aug 21, 2023

alistairsellar left a comment

Choose a reason for hiding this comment

alistairsellar Aug 30, 2023

Choose a reason for hiding this comment

alistairsellar Sep 27, 2023

Choose a reason for hiding this comment

zklaus commented Sep 1, 2023

alistairsellar left a comment

Choose a reason for hiding this comment

axel-lauer commented Sep 28, 2023

schlunma commented Oct 10, 2023

axel-lauer commented Oct 10, 2023

flicj191 left a comment

Choose a reason for hiding this comment

flicj191 May 9, 2024

Choose a reason for hiding this comment

rbeucher left a comment

Choose a reason for hiding this comment

axel-lauer commented Mar 28, 2023 •

edited

Loading