A Dataless Cube

[bjlittle]

✨ Feature Request

I think it's healthy to challenge established norms...

I want the ability to create a dataless cube. By this I mean the ability to create a hyper-space defined only by metadata i.e., no data payload

Once data is added to the cube, then the dimensionality is established and locked down, as we traditionally know and accept.

Motivation

Such hyper-spaces could be used in various ways e.g.,

• as a factory to manufacture fully formed cubes for test data
• as the target hyper-space in a regridding or interpolation transformation

I'm sure there are more concrete use cases... Please do share them on this issue if you know or any 🙏

Traditionally, there are many situations where a cube enforcing that it must have data is simply an inconvenience. Given the natural progression of model resolutions it seems "just wrong" to abuse dask to create lazy data that will never be used. It reeks of something not being quite right to me.

Let's do something about that 😉

Please up vote this issue if you'd like to see this happen 👍

Steps

• Complete work described in A Dataless Cube #4447 (comment)
• Write up remaining 'visions' for a dataless Cube in separate issues. Encourage the users that have upvoted this issue to vote on one/more of the new issues if that outstanding work is important to them
• Close this issue

[DPeterK]

@bjlittle supermegahypercubes! That is, a cube that describes how huge numbers of incoming datasets would tile together to make an n-dimensional hyperstructure - think, for example, of representing an entire model run in a single object. This would ideally be represented as a metadata-only cube, with individual data payloads very much fetched on demand only, given the vast quantities of data such an object would represent.

We've considered this idea from a variety of different perspectives in the Informatics Lab, and we think it has legs. We've also given the idea a bunch of different names, but supermegahypercubes is the best, most whimsical and original name we came up with for the concept 🙂

[pp-mo]

@bjlittle are you including here the idea that possibly only some of the data might be "filled", with some of it left unidentified.
So, that might be closer to the idea previously suggested which I think was maybe called a "hypercube", probably in the Informatics Lab ? IIRC that was certainly raised before but we never managed to get around to seriously considering it.
( @DPeterK I can't find an issue link for this -- maybe can you help ? )

P.S. as a name, for that idea at least, I think "hypothicube" is neater (though for language purists that should probably be "hypothecube" 😉 )

[edmundhenley-mo]

@bjlittle - re your concrete use-cases:
If useful to see some (~pedestrian, non hyp[er|o]cube-y) code-in-wild examples of target hyperspace for interpolation/regridding, I've got a couple here (sorry, only viewable internally@MO).
Almost certainly not optimal, but guessing poss still useful to see non-expert usage!

• Adding a np.zeros .data cargo when defining target cube for a model->model regrid, cargo immediately getting discarded when interpolation/regridding applied. Used here. In this case just a very small cargo, so not especially wasteful to create/discard, but can see that in other cases would be!
• Alternatively, a similar ~pointless NaNing of data here, for some obs->model comparisons, where the target cube (to eventually accept some interpolated observations) was based off another (model data). Subsequent wrangling of metadata to remove entries irrelevant in the obs data case.

[edmundhenley-mo]

@pp-mo - dunno re issue, but wonder if you're recalling the part-filled example in Jacob's hypotheticube article? Or poss another Informatic Lab article? (here's @DPeterK 's one on supermegahypercubes

[philip-brohan]

I feed streams of cubes through Machine Learning software (TensorFlow - TF). This requires throwing away the metadata and operating only on the data arrays, and then laboriously reconstructing metadata around the output data. It would be great to be able to cut a cube into data and metadata components, process them seperately and recombine them later.