Issue overview: finalize and verify extension, store and codec mechanisms

[jstriebel]

This is a high-level issue to track the progress around extensions, stores and codecs to finalize the v3 core spec. Finding the right abstraction levels for those is important, since it provides hints for implementors where to generalize concepts, which might make it easier in the long run to add future extension. In short: Having good extension points helps to grow support for extensions. Personally, I think the current status of the extensions is great, this is just to finalize those.

Please use or open the specific issues to discuss single points here. This is mostly to meta-discuss the overall strategy and overview, please feel free to recommend items that should be on this list, this just represents my current thoughts and neither exhaustive nor all strictly necessary (although I'll try to update it based on progress and comments):

Ideas for other extension points

Those are ideas that were floating around in meetings and issues, we might want to add those extension points

• WIP Global storage transformer extension mechanism. PR Global storage transformers #182

• Metadata conventions could help to standardize metadata fields. Metadata mostly comes and is used by

  1. the writing zarr-lib itself
  2. higher-level libraries and viewers (eg. GeoZarr, OME-NGFF, neuroglancer, webKnossos, …)
  3. the user.

  Having a specific section and/or document for metadata convention would foster cross-compatibility between producers and viewers (i. & ii.). I think this should not enforce anything on the zarr level, but is a guideline/convention which might be used implementations. This also popped up in Writer name field #139 and Revise how the domain of an array is specified #144. I'll try to draft a PR for this to have some basis for discussion.

  See ZEP 4

Finalize a spec per extension/store/codec:

To verify that all extension points are specified in full depth we might want to have finalized examples for each of those, also for stores and codecs. It might be useful to either include some of them directly into ZEP 1, or directly open separate ZEPs.

• Finalize data type extension (should be done in a different ZEP)
• Finalize one store spec,
  probably the file system store
  see Make filesystem store part of zep 1 #195.
• Finalize at least one codec, maybe simply all current ones at https://zarr-specs.readthedocs.io/en/latest/codecs.html
  See v3 Spec: Finalize current codecs #87, move codecs into separate (versioned documents), update urls #187
• Add v2 compatibility extension
• Clarify dependencies between extension points. E.g. must per-array-extensions be mentioned in the entrypoint metadata? Must data-types, storage transformers etc. be listed in the extensions of the array?
  See Why do storage transformers need "type" separate from "configuration" #191

Add templates

Once the points above are finalized we might want to extract templates from them to ease adding new extensions and facilitate homogeneous specs across those.

[jstriebel]

see also #89

[jstriebel]

This is mostly done, only #182 seems left over for ZEP 1