Consolidate cubes to make them much faster

[forman]

Is your feature request related to a problem? Please describe.

Opening larger cubes from OTC Object Storage or AWS S3 takes too long. Analysis has shown that this is caused mainly by the many HTTP requests made for

• .zarray and .zattrs metadata files, either by checking for existence or to load them;
• time coordinate chunks if the chunking for time is one (e.g. chunks={'time': 1, 'lat': 1000, 'lon': 1000) .

Describe the solution you'd like

"Consolidate" cubes after each append, insert, and overwrite of time slices using xcube gen. That includes

1. calling zarr's consolidate_metadata function;
2. combine all time chunks into a single so time coordinate variable is never chunked.

TODO

• a new CLI impl. module and public command consolidate <dataset> so we can consolidate existing cubes.
• a new API impl. module and public function consolidate_dataset
• call consolidate_dataset from xcube gen after creating or extending cubes
• in xcube serve, use xarray's open_zarr function with the consolidated=True flag.

Additional context

CyanoAlert cubes in OTC OBS take up to a few minutes to open, see #131 and xcube-dev/xcube-viewer#61

[forman]

consolidate --> optimze

[forman]

@AliceBalfanz, @dzelge: I believe we may/should now remove xcube prune and instead make pruning an option of xcube optimize. What do you think?

[dzelge]

I think, that sounds right.

…

------------------------ Helge Brockmann Consult Von: Norman Fomferra <notifications@github.com> Gesendet: Montag, 12. August 2019 14:49 An: dcs4cop/xcube <xcube@noreply.github.com> Cc: Helge Dzierzon <helge.dzierzon@brockmann-consult.de>; Mention <mention@noreply.github.com> Betreff: Re: [dcs4cop/xcube] Consolidate cubes to make them much faster (#141) @AliceBalfanz <https://github.com/AliceBalfanz> , @dzelge <https://github.com/dzelge> : I believe we may/should now remove xcube prune and instead make pruning an option of xcube optimize. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#141> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AKDZYIIUMOUB24IPPH6GESDQEFL2XANCNFSM4IJ5BMUQ> . <https://github.com/notifications/beacon/AKDZYIPJZAMWNWOBLEDVPADQEFL2XA5CNFSM4IJ5BMU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CNPMY.gif>

[AliceBalfanz]

I agree.

[forman]

Done.

[forman]

Reopened, as xcube server does not yet support the consolidated flag.

[forman]

Done