add gmet zarr dataset to intake cat

[jhamman]

xref: pangeo-data/pangeo-example-notebooks#5

[martindurant]

I am +1 for this.
However, the full dataset contains rather a lot of files and is slow to load the metadata. The chunksize of 14MB is OK order-of-magnitude but perhaps would be better at ~100MB. Perhaps this data-set also needs the metadata treatment in #309 , and a general agreement on how to cope with the many-files problem for the future (half-million files in this case).

[jhamman]

@martindurant - I can update the chunksize in the dataset if that would be useful.

[martindurant]

I certainly think that would help, and ~> 100MB is a good place to aim, perhaps bigger as the total dataset increases.

[jhamman]

Okay, I'll push the dataset again with larger chunk sizes. I'll update here when its ready.

[martindurant]

The catalog doesn't change, right?

[jhamman]

Right. I may remove the .zarr suffix though on the bucket. Thoughts?

[mrocklin]

Perhaps this data-set also needs the metadata treatment in #309

How does one do this? Do we need a best practices doc? Should this be upstreamed to XArray in some way?

…

On Fri, Jul 27, 2018 at 10:28 AM, Martin Durant ***@***.***> wrote: The catalog doesn't change, right? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#341 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszE8qe18EA1PAKQGv5GHbrVHvzhLuks5uK03QgaJpZM4ViOfF> .

[martindurant]

@mrocklin , right, it's strictly ad-hoc, we didn't come to a conclusion on how to consolidate the metadata in a way to causes least friction. My simplistic solution was to put a file .zmetadata in the top-level with the contents of all of the rest of the .z* files in the hierarchy; then zarr.Group looks for this key in the mapping. Following the discussion on metadata over on zarr, it could, for example, have been an additional entry in .zattrs in the top-level, and so stay within the spec.

[martindurant]

(but yes, it could have been a PR to xarray, that's part of the conversation in #309)

[martindurant]

I may remove the .zarr suffix though on the bucket. Thoughts?

I mildly side with keeping it, for people that browse the file hierarchy in the bucket.

[mrocklin]

OK. I'd like to make sure that if this is important that it doesn't get lost. It looks like nothing significant has happened on that issue for about a month. @martindurant do you have thoughts on the right way to come to some long-term conclusion on this issue? Should it be elevate to some upstream issue tracker?

…

On Fri, Jul 27, 2018 at 10:36 AM, Martin Durant ***@***.***> wrote: (but yes, it could have been a PR to xarray, that's part of the conversation in #309 <#309>) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#341 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszDScv4fYUDsSE7EzjCq_2krKr7d0ks5uK0-UgaJpZM4ViOfF> .

[martindurant]

We have zarr-developers/zarr-python#268 , which is also getting old.

[alimanfoo]

Happy to pick that discussion up again on the Zarr side if that would help.

…

On Fri, 27 Jul 2018, 18:55 Martin Durant, ***@***.***> wrote: We have zarr-developers/zarr-python#268 <zarr-developers/zarr-python#268> , which is also getting old. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#341 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qr0cj2pqCMQz-QHVMh6vNM68ZqgLks5uK1PzgaJpZM4ViOfF> .

[jhamman]

I rechunked the dataset. We're down to 16k chunks and up to ~80mb/chunk.