Find files for CMIP6 DCPP startdates

[sloosvel]

Before you start, please read our contribution guidelines.

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• This pull request has a descriptive title that can be used in a changelog
• Add unit tests
• Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
• If writing a new/modified preprocessor function, please update the documentation
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes
• If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.

---

A 'startdate' tag, that gets expanded in the same way as ensembles, can be added when calling some CMIP6 DCPP experiments that follow the scheme:

{short_name}{mip}{dataset}{exp}{startdate}-{ensemble}_{grid}*

The paths sometimes include the startdate, but in other cases they do not, so both cases are considered:

• ../{exp}/{startdate}-{ensemble}/... (as in Jasmin)
• ../{exp}/{ensemble}/...

@jvegasbsc is there any other that should be considered?

Closes #632

[jvegreg]

Nice work! I just added a couple of tests for the replace_tags function

[jvegreg]

@remi-kazeroni, can you please test it in DKRZ to see if we need some extra work to support it. If you or any other volunter don't have time to test it, we can merge it so at least we can read DCPP data in Jasmin

[remi-kazeroni]

@remi-kazeroni, can you please test it in DKRZ to see if we need some extra work to support it. If you or any other volunter don't have time to test it, we can merge it so at least we can read DCPP data in Jasmin

Yes sure, I'll have a look at this PR this week

[zklaus]

Note that what you call startdate here is actually part of the CMIP6 DRS as sub_experiment. See CMIP6 Global Attributes, DRS, Filenames, Directory Structure, and CV’s, particularly p. 13, DRS components and p. 14, filename template. This also makes reference to the applicable CV in CMIP6_sub_experiment.json.

Perhaps it would be prudent to adopt the same terminology. That would also open the functionality up to other MIPs that might have sub-experiments in the future.

The first linked document also tells us what the directory structure should be (p.17, directory structure template), but of course we can't force every data center to conform.

[remi-kazeroni]

Nice work! I have tried on various combinations of datasets with and without sub_experiments and all of them can be read as expected. I just have a few general comments, feel free to ignore if not relevant:

1. Do you think it would be useful to document these changes? The sub_experiment key could be added here
2. The start_year and end_year are not needed when a sub_experiment is provided in the recipe. Instead, the decadal time bounds are used (e.g. start_year: 2000 and end_year: 2010 for sub_experiment: s2000-r1i1p1f1). One can still consider a shorter time range (e.g. 2005-2008) but in this case filenames contain the whole decadal time range instead of the shorter range (e.g. plots/*_dcppA-hindcast_s2000-r1i1p1f1_tas_2000-2010.png and not plots/*_dcppA-hindcast_s2000-r1i1p1f1_tas_2005-2008.png). Is that expected? Perhaps this example is not relevant in practice...

[jvegreg]

The start_year and end_year are not needed when a sub_experiment is provided in the recipe. Instead, the decadal time bounds are used (e.g. start_year: 2000 and end_year: 2010 for sub_experiment: s2000-r1i1p1f1). One can still consider a shorter time range (e.g. 2005-2008) but in this case filenames contain the whole decadal time range instead of the shorter range (e.g. plots/_dcppA-hindcast_s2000-r1i1p1f1_tas_2000-2010.png and not plots/_dcppA-hindcast_s2000-r1i1p1f1_tas_2005-2008.png). Is that expected? Perhaps this example is not relevant in practice...

Without changing the way we manage the time range, handling the start and end year will be a nightmare for the full hindcasts, as we will have to specify it manually for all years from 1979 to 2010 or so. The ability to use s(1979:2010) will be useless.

We have a pending issue to deal with the time range #345 and I remember another suggestion asking for a way yo specify relative ranges (first 20 years, last 50, all available...) I think we should do this rework in a separated pull request.

[zklaus]

The sub-experiment should really be completely independent from the startrange and endrange. It only tells you something about the initialization of the experiment. Every run is 10 years long, but there is a new run started for every year. This means that there are many different sub-experiments covering the same years and comparing these is one of the main interests of the DCPP mip.

Consequently, we should expect it to be common to find dataset blocks like

datasets:
  - {sub-experiment: s1960, start_year: 1965, end_year: 1970}
  - {sub-experiment: s1962, start_year: 1965, end_year: 1970}
  - {sub-experiment: s1963, start_year: 1965, end_year: 1970}
  - {sub-experiment: s1964, start_year: 1965, end_year: 1970}

[sloosvel]

The sub-experiment should really be completely independent from the startrange and endrange.

I think it already is. This finds and loads all the files available for a certain sub-experiment value. And later it adds the overall start_year and end_year. It's true that you cannot subset it as @remi-kazeroni says, but I think people normally load all years available anyway.

Consequently, we should expect it to be common to find dataset blocks like

datasets:

• {sub-experiment: s1960, start_year: 1965, end_year: 1970}
• {sub-experiment: s1962, start_year: 1965, end_year: 1970}
• {sub-experiment: s1963, start_year: 1965, end_year: 1970}
• {sub-experiment: s1964, start_year: 1965, end_year: 1970}

I am a bit lost because that's what I was trying to avoid: having 50 lines in a recipe to call a single experiment with multiple subexperiments.

[zklaus]

I am a bit lost because that's what I was trying to avoid: having 50 lines in a recipe to call a single experiment with multiple subexperiments.

Why?

[sloosvel]

Because everything belongs to the same dataset-experiment pair, which is what every new line in a recipe represents. It also makes the recipe easier to read.

And the end goal will be to be able to deal with all those sub-experiments as one in a single cube with a new dimension. So it's like a single dataset.

[zklaus]

I think in the context of the DCPP mip, every sub-experiment should be seen as its own experiment. You don't always (or even generally) want to look at all sub-experiments at the same time.

[remi-kazeroni]

datasets:

* {sub-experiment: s1960, start_year: 1965, end_year: 1970}

* {sub-experiment: s1962, start_year: 1965, end_year: 1970}

* {sub-experiment: s1963, start_year: 1965, end_year: 1970}

* {sub-experiment: s1964, start_year: 1965, end_year: 1970}

I am a bit lost because that's what I was trying to avoid: having 50 lines in a recipe to call a single experiment with multiple subexperiments.

But I think this can already be done in one line with this PR using: {sub-experiment: s(1960:1964), start_year: 1965, end_year: 1970} (assuming 1961 is used as well).
I tested that with the examples/recipe_python.yml and only the data between start_year and end_year are used and plotted. But the filenames in the plots dir are in the form *dcppA-hindcast_s1964-r1i1p1f1_tas_1964-1974.png while one would expect *dcppA-hindcast_s1964-r1i1p1f1_tas_1965-1970.png

[zklaus]

Sure, there are many ways to specify these. The point is neither should we manipulate start and end because of sub-experiment, nor sub-experiment because of start and end.

[sloosvel]

So which changes in this pull request do you not approve of?

[zklaus]

So which changes in this pull request do you not approve of?

In the review above I tried to mark all the places that this touches. All of those remarks really are about the same issue.

[sloosvel]

Good to know! I'll try to see what can be done about them.

[sloosvel]

If the new changes are of your liking, I will add the tests and everything else that's missing.

[zklaus]

There is one comment from the previous review left to address.

Now, this PR seems to introduce new functionality in terms of all_years. That appears to be completely unrelated to DCPP, so I suggest putting it into a different PR. That way we will be able to wrap this one up more quickly and the functionality (that will be appreciated by others for other experiments!) will get better visibility.

[sloosvel]

That appears to be completely unrelated to DCPP,

The DCPP functionality that we need at the department requires it. This pull request without it does not help much our use case.

Would it be that bad to include it as a first step, and then refine it for other experiments in another pull request?

[zklaus]

I personally think that the changes will be available quicker if split up into two PRs because it makes the review easier and the changes more self-contained. If you really prefer to have it together we can continue here; in that case please add documentation of the new functionality in a place where it can be found. I think the functionality would already work for other experiments, no?

[zklaus]

Ok to merge for me. Nice work!

[sloosvel]

Great! @ESMValGroup/esmvaltool-coreteam anyone with the time to merge it?

[valeriupredoi]

good stuff @sloosvel 🍺