Simple support for selective netcdf loading.

[pp-mo]

🚀 Pull Request

Description

An initial heads-up on a really simple way of speeding up netcdf loads ...
... with files of many variables, as in #4134

This is actually much simpler than I imagined, but ..
Blockers to completion

• probably needs discussion
• intended usage might require to select multiple data-vars
• not sure how to test this

Background...

Following #4135, ESMValTool devs are reporting that iris loading is still too slow, where you want only one from a whole lot of diagnostics.

This example follows what we did for UM files in similar circumstances.
It's notable that, in creating a iris.fileformats.cf.CFReader, we are still doing a whole-file analysis, that includes the unwanted data-variables.
I actually don't think you can avoid that, as only context will distinguish a CF data-variable from an aux-coord.
However, the cost of this is not huge. I am finding <1sec for the testfile mentioned in #4134
( which is : ~250mB, ~300 variables of content float[1 * 79 * 143 * 144] )

Some sample timings:

Using testfiles with many identical (small) variables:
n-vars : timings without // with fix
1 : 0.04 // 0.01 [loading 1 of N named variables]
10 : 0.14 // 0.02
30 : 0.45 // 0.03
100 : 1.70 // 0.07
300 : 8.19 // 0.40
(314) : 45.36 // 0.59

case (314) is based on the testfile mentioned in #4134
( I suspect it may be slower than the '300' because the variables data is larger?? WIP )
with the code like

cube = iris.load_cube(
    'Iris_multivar_data_file.nc',
    NameConstraint(long_name='Air Surface Temperature'))

---

Consult Iris pull request check list

[trexfeathers]

I've stared at this for a while now, and it makes sense.

My main question is: what does this look like beyond MVP? There are obviously a lot of limitations and hard-coding in this implementation; how would you make it more generalisable? I.e. multiple constraints, non-NameConstraints, less code duplication? Since we don't know when we might have time to expand the functionality, it would be good to make sure the merged code is something that can be iterated on, rather than something that would need replacing.

You can ignore my question if this is far as we're expecting to take this functionality in the foreseeable future.

[pp-mo]

what does this look like beyond MVP?

Well, first we need to check that this is addressing the immediate needs of the ESMValTool devs, otherwise it's not worth it.

A simple list of constraints acts with an "OR"-like behaviour, so that is mostly a non-starter for a speedup solution.
The simplest exception to that is provided for in the PP implementation, where we handle a list of constraints all of which are of the form AttributeConstraint(STASH=<selected>).
You could in principle extend that to cases where all the top-level constraints in the list are formed as '&' -combinations (i.e. ConstraintCombination), and one of those is a type we can "translate".
But in the PP case we never even got that far.

Apart from that, I think in this case we can work out how to do the equivalent of a Constraint(name=string), allowing that we decode a possible STASH attribute on the variable.
So I think that's about all that is technically possible + we need anyway to first decide what is actually useful before committing more time.

[trexfeathers]

Well, first we need to check that this is addressing the immediate needs of the ESMValTool devs, otherwise it's not worth it.

Yup, makes sense not to invest time thinking about the future yet. In that case do you think it would be worth including a note or todo in the docstring to indicate that it's a bit of an experiment? If it's valuable: we'll need to re-write it more generally, if it's not: we'll need to back it out.

Indeed, I'd quite like if we could move translate_constraints_to_var_callback to the experimental module.

[pp-mo]

move translate_constraints_to_var_callback to the experimental module.

Ok I don't think there's any point doing that directly, as it isn't a public thing
(though I can see it should maybe have a "_" -- will do that !)

My feeling is that this is (should be) an implementation shortcut guaranteed to have no effect on correctness I.E. the final result is the same in all cases.
Equally importantly, we can then provide it with no visible user control.
So if you can keep it like that, then it doesn't really matter which cases it covers and which not,
( except whether it addresses a performance need ).

[pp-mo]

Thanks for inputs @trexfeathers @bjlittle
Will address those points in due course.
But, I'm going to hold fire on this until it's clear whether it adresses the ESMValTool devs' need.

[pp-mo]

Some further practical discussion at ESMValGroup/ESMValCore#1180 ESMValGroup/ESMValCore#1153 (comment) (the discussion has been moved to its own issue).

In my reply , I tried to explain how the existing solution here is only a quick hack.

As-is, it only supports a constraint expression which is a single NameConstraint.
But as noted above, we already know how to support a list of several of those (done for PP).

Some possibly-useful further cases :

• We should probably support a simple string as constraint --> Constraint(name) --> match any of : standard/long/var-name or STASH-attribute
  • (all-of which can still be found in raw netcdf data, so eminently possible)
• a list of single constraints, of the types we can handle (NameConstraint or Constraint(string), aka "translatable constraints")
• We could also support, for example, any &-combinations involving a single translatable-type-Constraint as one of the 'terms'.
• We can also do AttributeConstraints if wanted.

But the approach does not generalise fully -- i.e. some types of expression which "could" be translated (+ therefore speeded up) will remain 'opaque' to this type of analysis and it won't be possible to do everything.
Most notably, anything involving a selection-function is obviously "opaque", and that includes 2 important common cases :

• anything with a cube_func option
• a CoordConstraint testing for a value-range, e.g. coord=lambda cell: min <= cell < max

Likewise, in any case where we are given a list of constraints, we need to be able to 'translate' everything on the list, or we can't select on data-variable at all.

[zklaus]

For us, the main use case is to load one variable from a multi-variable file. I think that is really all we need.

[pp-mo]

Status report

EMSValTool people have confirmed that they only need the simplest case of this (one named var from one file)
Reviewing of outstanding work, as we want to get this into Iris 3.1.0

1. respond to outstanding (small) review comment, and any further
2. fix for multiple constraints, as it is in PP code + should be easy (possibly combine in common code ??)
3. likewise, should be easy to support Constraint(name) form in addition to NameConstraint
   • so should probably do that to avoid confusion (as NameConstraint is a bit 'specialist').
4. add whatsnew

(2) and (3) can be dropped if there is any difficulty or hurry

I (@pp-mo) can tackle this in ~2 weeks when I return from holiday,
or someone else can do it ?

[lbdreyer]

I have taken this on whilst @pp-mo is on holiday.

Changes I have made:

• Rebased onto latest iris/main (this PR was originally written before @pp-mo's pyke removal)
• Moved the iris._constraints import
• Added a whatsnew
• Added unit and integration tests
• Added support for NameConstraints with a var_name
• Moved iris.fileformats.netcdf.translate_constraints_to_var_callback to be a private function so that it is not a part of our public api.

[lbdreyer]

Rebased to resolve merge conflict now that #4206 has gone in

[bjlittle]

@lbdreyer Awesome! Thanks for helping to nudge this PR across the line 🍻

Just a couple of comments to service, then we're good to go 🚀

[lbdreyer]

@bjlittle Thanks for your review! I believe I have now resolved the review actions you raised, so could you have a final check?