Handle IPSL-CM6 #1153

senesis · 2021-05-31T09:57:59Z

Description

Related to issue #1143. Allow to handle IPSL-CM6 data formats and file naming conventions.

Composed of :

the addition of two files : a fix module and a variable mapping file (extra_facets, see Add basic support for variable mappings #1124 )
the update of config-developer.yml and config-user.yml
update of the documentation

The feature won't actually work without #1124 , but the proposed PR is compatible with the main branch before #1124 merge, in order to allow for immediate merge, and so to avoid bottlenecks. #1124 allows for finding the right file and the right variable in file when using ISPL-CM6 data.

Some documentation on the handling of native models has been introduced.

Performance in loading variables from IPSL's multi-variable datafiles with Iris is poor, even with the Iris 3.0.2 fix on that subject. This is why pre-selecting using CDO in a fix_data method is implemented. However, because CDO licence (GPL v2) is not compatible with ESMValTool licence policy, this feature is not activated by default, and no explicit dependency is introduced. The feature can be activated through a variable mappings file (see #1124), at those sites which accept to be dependent on CDO.

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🧪 The new functionality is relevant and scientifically sound : scientific relevance is explained in IS-ENES3 WP9.5 2nd subtask : Enhancing the use of the ESMValTool for model development
🛠 This pull request has a descriptive title and labels
🛠 Code is written according to the code quality guidelines
🧪 and 🛠 Documentation is available : a slight change of doc structure was introduced (renaming quickstart section 'Finding data' to 'Input data') , with relevant changes in quickstart/configure and develop/fixing_data
🛠 Unit tests have been added
🛠 Changes are backward compatible
🛠 Any changed dependencies have been added or removed correctly : see above about CDO
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful : Codacy check failures will be removed by merge of Add basic support for variable mappings #1124 , except for the use of subprocess. Commit checks are no more activated, for an unknown reason

* First attempte * Do not require start and end years, add them later * Correct condition * Avoid key error in fx variables * Consider two possible paths * Fix function name * Fix variable name * Avoid duplicates in filename * Add test for startdate expansion * Add test for the replace tags method * Rename tag * Add documentation * Allow to load subexps per timerange or as a whole * Fix condition * Remove 'all_years' functionality * Fix conditions * Fix flake * Remove whitespace Co-authored-by: Javier Vegas-Regidor <javier.vegas@bsc.es>

…grid (#507) Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl> Co-authored-by: Stef Smeets <s.smeets@esciencecenter.nl>

* sos and siconc fixed * tests added * test fixed * fix flake8 * fix flake8 * fix codacy issue

* pin cf-units * pin cf-units * fix test * fix test

jvegreg

One question: how can I access to a data sample so I can properly test it?

esmvalcore/cmor/_fixes/native6/ipsl_cm6.py

senesis · 2021-05-31T17:06:32Z

One question: how can I access to a data sample so I can properly test it?

An example of multi-variable file for IPSL-CM6 is available here (on my Google Drive)
Data is not enough : this PR anticipates on a merge of #1124, I tested the whole on a local branch which included #1124. I can push such a branch to the repository if you would like to run a test

senesis · 2021-06-01T14:05:11Z

@alistairsellar : you may wish to know that this PR exists

zklaus · 2021-06-02T14:19:37Z

Quick side note on the slow loading: I am just talking to some iris developers and it seems they are hitting the exact same problem, also while dealing with an XIOS generated multi-variable output file for the next model generation. There doesn't seem to be an issue about it yet, but I forwarded your example file and will try to add the issue here when it becomes available.

senesis · 2021-06-02T17:01:55Z

Quick side note on the slow loading: I am just talking to some iris developers and it seems they are hitting the exact same problem, also while dealing with an XIOS generated multi-variable output file for the next model generation. There doesn't seem to be an issue about it yet, but I forwarded your example file and will try to add the issue here when it becomes available.

I reported the problem quite early as an Iris Issue, and provided the file plus a notebook showing the problem by profiling an Iris run, and provided a follow up in my original issue on this subject, which basically says that even with a recent fix in Iris V3.0.2, the load is still slow (figures availabe on demand). Do you mean that the Iris developers have some new idea here ?

pp-mo · 2021-06-03T17:29:44Z

@senesis even with a recent fix in Iris V3.0.2, the load is still slow

Following discussion with @abooton @zklaus , we've been looking into another possible Iris improvement : to avoid translating data-variables which are not wanted in a load.
A crude demonstration shows a very considerable speedup for loading just one selected cube.

I rather struggled to understand exactly what you need to load here, but I'm assuming may be just a single data-variable,
based on this comment and the CDO use proposed in this PR.
If it's actually more complicated than that, we can probably still adopt the general approach but it will need to be more sophisticated.

senesis · 2021-06-05T13:41:05Z

Commit #896764b bring this PR in line with PR #1124 at commit 50afc8b. I may complement this PR at short notice if needed, including switching to the option of supporting IPSL-CM6 through a dedicated project. I will fix codacy tests diags after the merge of #1124 in main, because a number of them are due to the anticipation on fix.extra_facets existence

bouweandela · 2021-06-07T08:13:13Z

including switching to the option of supporting IPSL-CM6 through a dedicated project

That would be good, otherwise you will not be able to use the IPSL model data at the same time as any of the other data supported by the native6 project.

zklaus · 2021-06-10T14:11:30Z

For now, I just reduced the number of changes by reverting to the previous formatting in unrelated sections. As you can see from the diff, this is much easier to digest. Note that I don't claim that this formatting is better in any way. It just makes it easier to distinguish substantive changes, that we need to proofread, from the rest.

The main things for this were:

Remove trailing spaces (perhaps something your editor may support you with)
Turn tabs into spaces
Restore the original line-breaks where possible

I will now do a proper review of the documentation changes.

zklaus

Ok, finished the doc review. Only minor comments.

doc/develop/fixing_data.rst

doc/quickstart/find_data.rst

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

zklaus · 2021-06-10T15:34:25Z

I made a small improvement to the formatting.

I also have one more question: The atmospheric variables are now separated according to CMIP vs IPSL names. Does this not apply to the other realms? Also, there are a number of very long #============ lines in the facets file. Do we need them? If you want to keep them, could we shorten them, maybe to 79 chars total line length or similar?

senesis

Fine

doc/develop/fixing_data.rst

…Core into Handle_IPSL_CM6

@bouweandela

The review comments were addressed and we @bouweandela and I discussed offline how to move forward.

zklaus · 2021-06-15T14:44:11Z

I have moved the discussion on Iris loading times and CDO to the new issue #1180. Let's continue there.

senesis · 2021-06-19T09:35:38Z

By the way : thanks a lot for the support on this PR and the related issue ; nice to have that yet in V3.2

Klaus Zimmermann and others added 7 commits May 12, 2021 11:38

Add basic support for variable mappings

8b8dd93

Add first era5 mapping

9f2794f

Skip regridding if the target grid is almost identical to the source …

d846877

…grid (#507) Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl> Co-authored-by: Stef Smeets <s.smeets@esciencecenter.nl>

Fixes for sos and siconc of BCC models (#1090)

c20e5c1

* sos and siconc fixed * tests added * test fixed * fix flake8 * fix flake8 * fix codacy issue

Pin cf-units and fix tests (cf-units>=2.1.5) (#1140)

1f788c2

* pin cf-units * pin cf-units * fix test * fix test

Handle IPSL-CM6 (the feature won't actually work without #1124)

8fc038f

senesis added the enhancement New feature or request label May 31, 2021

senesis added this to the v2.3.0 milestone May 31, 2021

senesis requested a review from jvegreg as a code owner May 31, 2021 09:57

senesis self-assigned this May 31, 2021

senesis requested review from bouweandela and zklaus May 31, 2021 09:59

jvegreg reviewed May 31, 2021

View reviewed changes

esmvalcore/cmor/_fixes/native6/ipsl_cm6.py Outdated Show resolved Hide resolved

esmvalcore/cmor/_fixes/native6/ipsl_cm6.py Outdated Show resolved Hide resolved

senesis added 2 commits May 31, 2021 19:11

class Huss inherits from cass Tas. Also : Fix codacy diags.

3b2a05f

Replace os.system() by subprocess.run()

9303c90

Fix flake8 diags

8d55ee8

This was referenced Jun 3, 2021

Simple support for selective netcdf loading. SciTools/iris#4175

Closed

Simple support for selective netcdf loading. SciTools/iris#4176

Merged

senesis added 3 commits June 5, 2021 09:12

var_mapping -> extra_facets

eacd764

Rename _config/variable_details to _config/extra_facets

684953e

Fix doc re. lack of 'output_file as a dict', and choice of native6

896764b

Fix codacy diags in ipsl_cm6.py

6590586

zklaus force-pushed the Handle_IPSL_CM6 branch from 0d741c7 to 89f45b9 Compare June 10, 2021 14:05

zklaus requested changes Jun 10, 2021

View reviewed changes

senesis and others added 10 commits June 10, 2021 17:05

Update doc/develop/fixing_data.rst

6b085e5

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

36dc526

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

36a47fe

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

0625016

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

ad3a0e4

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

f6c81b5

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/develop/fixing_data.rst

c785b97

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/quickstart/find_data.rst

e98cac6

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Update doc/quickstart/find_data.rst

862b362

Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Minor formatting improvements

4112fa9

Organize mapping file in each realm in two sections (CMIP6 and IPSL)

d7a4471

senesis commented Jun 10, 2021

View reviewed changes

doc/develop/fixing_data.rst Show resolved Hide resolved

Merge branch 'Handle_IPSL_CM6' of ssh://github.com/ESMValGroup/ESMVal…

2c2e2f5

…Core into Handle_IPSL_CM6

zklaus approved these changes Jun 11, 2021

View reviewed changes

zklaus merged commit 68d9ef8 into main Jun 11, 2021

zklaus deleted the Handle_IPSL_CM6 branch June 11, 2021 12:59

This comment has been minimized.

Sign in to view

zklaus mentioned this pull request Jun 15, 2021

Remove use of CDO for selecting the right variable in native model fixes #1180

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle IPSL-CM6 #1153

Handle IPSL-CM6 #1153

senesis commented May 31, 2021 •

edited

Loading

jvegreg left a comment

senesis commented May 31, 2021 •

edited

Loading

senesis commented Jun 1, 2021

zklaus commented Jun 2, 2021

senesis commented Jun 2, 2021

pp-mo commented Jun 3, 2021

senesis commented Jun 5, 2021

bouweandela commented Jun 7, 2021

zklaus commented Jun 10, 2021

zklaus left a comment

zklaus commented Jun 10, 2021

senesis left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zklaus commented Jun 15, 2021

senesis commented Jun 19, 2021

Handle IPSL-CM6 #1153

Handle IPSL-CM6 #1153

Conversation

senesis commented May 31, 2021 • edited Loading

Description

Before you get started

Checklist

jvegreg left a comment

Choose a reason for hiding this comment

senesis commented May 31, 2021 • edited Loading

senesis commented Jun 1, 2021

zklaus commented Jun 2, 2021

senesis commented Jun 2, 2021

pp-mo commented Jun 3, 2021

senesis commented Jun 5, 2021

bouweandela commented Jun 7, 2021

zklaus commented Jun 10, 2021

zklaus left a comment

Choose a reason for hiding this comment

zklaus commented Jun 10, 2021

senesis left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zklaus commented Jun 15, 2021

senesis commented Jun 19, 2021

senesis commented May 31, 2021 •

edited

Loading

senesis commented May 31, 2021 •

edited

Loading