Conformant ZarrV3 codecs and fill values

[ghidalgo3]

This change serializes the ZArray codec attributes filters, compressor, and order according to the ZarrV3 specification, along with the default fill value of the array. This should hopefully allow a theoretical ZarrV3 reader to read a zarr store produced by VirtualiZarr, one day.

I would consider this a breaking change because any existing VirtualiZarr stores that used the "before" format shown here would be unreadable now, but given that those metadata files were invalid zarrv3 anyway I'm not sure if it's worth handling that at possibility at read time. If you think it necessary, reviewer, I will implement it.

I'm also not too sure if adding the transpose and bytes codec unconditionally is necessary. I think readers can assume that no transpose codec means no data transposition obviously, but it's unclear to me if the codec pipeline must declare either one of:

• https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html or
• https://zarr-specs.readthedocs.io/en/latest/v3/codecs/bytes/v1.0.html

Also, what happens if a source file uses a codec that is not one of the specified codecs of ZarrV3? Does that mean the file cannot be represented in ZarrV3? Seems rather onerous.

It may make sense to also change _check_same_codecs to use the codec_pipeline list declared in this PR.

Before

{
    "attributes": {},
    "chunk_grid": {
        "configuration": {
            "chunk_shape": [
                2,
                3
            ]
        },
        "name": "regular"
    },
    "chunk_key_encoding": {
        "configuration": {
            "separator": "/"
        },
        "name": "default"
    },
    "codecs": null,
    "data_type": "int64",
    "dimension_names": [
        "x",
        "y"
    ],
    "fill_value": null,
    "node_type": "array",
    "shape": [
        2,
        3
    ],
    "storage_transformers": [
        {
            "configuration": {
                "manifest": "./manifest.json"
            },
            "name": "chunk-manifest-json"
        }
    ],
    "zarr_format": 3
}

After

{
    "attributes": {},
    "chunk_grid": {
        "configuration": {
            "chunk_shape": [
                2,
                3
            ]
        },
        "name": "regular"
    },
    "chunk_key_encoding": {
        "configuration": {
            "separator": "/"
        },
        "name": "default"
    },
    "codecs": [
        {
            "configuration": {
                "order": [
                    0,
                    1
                ]
            },
            "name": "transpose"
        },
        {
            "configuration": {},
            "name": "bytes"
        },
        {
            "configuration": {
                "level": 1
            },
            "name": "gzip"
        }
    ],
    "data_type": "int64",
    "dimension_names": [
        "x",
        "y"
    ],
    "fill_value": 0,
    "node_type": "array",
    "shape": [
        2,
        3
    ],
    "storage_transformers": [
        {
            "configuration": {
                "manifest": "./manifest.json"
            },
            "name": "chunk-manifest-json"
        }
    ],
    "zarr_format": 3
}

• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst

Issues:

• Transposition via codec #163
• Is the compressor type right?  #94

[TomNicholas]

Also, what happens if a source file uses a codec that is not one of the specified codecs of ZarrV3? Does that mean the file cannot be represented in ZarrV3? Seems rather onerous.

I'm going to tag @sharkinsspatial here who I think has thought about this more than I have

[TomAugspurger]

It seems like the default fill value for float is 0.0:

import zarr
import json
store = zarr.store.MemoryStore(mode="w")
z = zarr.empty((1, 1), store=store)
z[:]

array([[0.]])

(I'm not sure where on the Array / Store / Other object that information lives.)

It'd be nice if zarr-python had this as a constant that we could reuse. Would that make sense, or is there some reason not to I'm missing?

[TomAugspurger]

I also see a reference to NaN as a default at https://github.com/zarr-developers/VirtualiZarr/pull/193/files#diff-f5a7b84b3378d903e91ebf06f2db06dca5ad55d12e7c3bf8537a9b9bb1c4cfa0R361 (present on main).

[ghidalgo3]

I think the specification doesn't require a specific number, just that it not be null. See the note at the bottom of the fill_value section https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#fill-value . However zarr-python does default to 0 eventually and not NaN https://github.com/zarr-developers/zarr-python/blob/37a8441c20dae3b284803bb1b0d2e6c8f040fb3e/src/zarr/array.py#L231C9-L235C31 . I may have had some trouble with the unit tests, but I think it's better to be as similar as possible to zarr-python, I'll change the defaults to 0s.

[TomNicholas]

I think this could also be deferred to a later PR, especially if the true solution is to make it clearer what the default is upstream.

[TomAugspurger]

Looks good to me overall. One question about the default compressor in the new codecs pipeline.

[ghidalgo3]

Thanks @t-mcneely for pairing on this with me! I'll write up the results of our PoC with GOES data. Sorry for the messy commit and revert, we accidentally merged Sean's HDF5 branch to make reading GOES HDF5 files work.

[TomNicholas]

Thanks for going into these weeds guys! This is amazing. Just a couple of extremely minor comments then I will merge it.

[TomNicholas]

Thanks @ghidalgo3 !!