HDF5/NetCDF4 compatibility

[LDeakin]

I want to read HDF5/NetCDF4 files as virtual Zarr arrays with zarrs. I had a proof-of-concept for this with virtualizarr + #79 but ran into the following issues:

• zarrs not yet supporting a number of HDF5/NetCDF 4 codecs
• Invalid Zarr V3 metadata from virtualizarr: Zarr V3 metadata fixes zarr-developers/VirtualiZarr#248
• virtualizarr (and zarr-python etc) not supporting variable shaped chunks

TODO:

• Add extra codecs to zarrs for more HDF5/NetCDF4 support. For example (from Add more filters JuliaIO/Zarr.jl#154):
  • Fletcher32
  • FixedScaleOffset
  • Quantize
  • Shuffle
  • Delta
• Revive Experimental chunk manifest support #79 if virtualizarr ends up moving forward with it. icechunk manifests may replace it
• Wait for virtualizarr to support zarr-python V3 for correct metadata
  • Zarr reader zarr-developers/VirtualiZarr#271
• Wait for virtualizarr to support variable shaped chunks
  • This may require more effort on Zarr V3 standardisation, it is unclear if the current ZEP will proceed as-is

[TomNicholas]

FYI:

Revive #79 if virtualizarr ends up moving forward with it. icechunk manifests may replace it

Currently I think all the people interested are working on icechunk rather than the original chunk manifest in zarr idea. So you should assume that's the direction things will go in.

Wait for virtualizarr to support zarr-python V3 for correct metadata

This could be done very soon (and is something you could help with if you were interested!)

Wait for virtualizarr to support variable shaped chunks

This is much more complicated and will likely take longer. Especially if you want to wait for it to actually be in the specification rather than just experimentally in the python implementation.

[LDeakin]

Thanks for the clarifications @TomNicholas.

This is much more complicated and will likely take longer. Especially if you want to wait for it to actually be in the specification rather than just experimentally in the python implementation.

Just experimentally in the python implementation is all I need, the spec is stalled anyway. zarr-developers/zarr-python#2536 looks supportable, but I appreciate the simplicity of the current ZEP.