doc(v3): add v3 roadmap and design document

[jhamman]

This PR adds the V3 roadmap / design document that we have been discussing in #1569 and as part of the refactor working group (#1480). Comments on this document are still welcome.

[jeromekelleher]

This is a very helpful document. I think the proposed design, particularly in terms of making the SyncX classes use the Async ones is great.

My main input is around documenting the extent of expected code breakage, and giving downstream devs advice on how to keep their code running when v3 drops.

One thing I'm also confused about is whether on-file changes for z3 format are orthogonal to the code/API changes for zp3.0? Will zv2 on-file arrays be supported indefinitely?

I think these issues are important to be clear about up front, because they do inform later design choices.

[jeromekelleher]

Mentioning codecs here makes me wonder what the relationship with numcodecs is going forward. If isolating codecs is part of the zp 3.0, perhaps this should be discussed explicitly?

The point below about entry_points and [zarr.codecs] hits the same point.

[jhamman]

@normanrz mentioned the idea about writing a set of codec that would wrap numcodecs existing compressors/filters/etc. That would likely be the cleanest way to continue using numcodecs going forward. With the entrypoint api for codecs, the wrapper class could actually live in numcodecs and simply be registered as an entrypoint that follows the protocol. See #1588 for a start on this api.

[jhamman]

This also came up in today's refactor WG call. @normanrz has given this more thought than I have. Perhaps he can share a bit more on what you think is needed here?

[normanrz]

The Zarr 3 spec is pretty explicit about what codecs are in spec to ensure compatibility among implementations. That set of codecs is now implemented directly in zarr-python (with use of numcodecs).

However, it is desirable to experiment with other codecs in zarr-python and make use of numcodecs. I think there are 2 ways of doing this:

1. We could make generic adapters for numcodecs to fit the zarr codecs API (ie. ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec). That would also be useful to code v2 compatibility.
2. We could also introduce a new API in numcodecs that presents all codecs using the Zarr codec API. Not sure what the appetite for such a change on the numcodecs side is.

I am a bit concerned about the codec metadata that gets written to the jsons because most codecs in numcodecs are not in the Zarr v3 spec (or with different metadata). Maybe we should actually mark them as numcodecs codecs:

"codecs": [{"name": "numcodecs", "configuration": {"id":"blosc", "cname": "zstd", ...}}]

[MSanKeys963]

Thanks for raising this, @jeromekelleher, and the explanation here, @jhamman and @normanrz.

I was wondering what would be the ideal way to add a new codec (if and when we plan to add any) in Zarr-Python V3.0?
And what would be the best way forward out of the two options if we consider the possibility of extending the existing codecs in the V3 specification?

Also, it'd be great if we could get @jakirkham views on this.

[normanrz]

To add new codecs to zarr-python, developers can subclass the base classes (e.g. ArrayBytesCodec, ArrayArrayCodec, BytesBytesCodec) and register the codec using the [zarr.codecs] entry point in their pyproject.toml or programmatically call zarr.v3.codecs.registry.register_codec.

[jeromekelleher]

I'd register a -1 on adding a dependency on attrs at this point - the builtin dataclasses module does most of the useful things. (My personal theory is that installation problems grow like O(n^2) where n is the number of dependencies - so only each additional dependency gets harder to justify adding!)

[d-v-b]

I'm sympathetic to this perspective @jeromekelleher. I could imagine switching from attrs to dataclasses (or even just vanilla python classes :) ) once the API changes have cooled down.

[normanrz]

I agree that we should remove attrs. However, for development right now we can probably move faster when keeping it in right now and later do one concerted effort to replace it.

[jeromekelleher]

If this is really the extend of v2 breakage (super minor!), I'd suggest toning down the language at the top about zp3.0 introducing breaking changes from zp2.x. Something like "Zarr-python 3.0 introduces some minor breaking changes, mostly around compatibility with h5py"

[jeromekelleher]

Also below "Zarr-Python 3.0 will introduce a number of new APIs and breaking changes to existing APIs." makes me nervous as someone with a lot of code depending on zp 2.0 APIs - addressing the extent of expected breakage, and what people can do to prepare for it at the top of the document would be great.

[rabernat]

This is really important point @jeromekelleher! The doc says there will be breaking changes to APIs, but doesn't specify what those changes will be. In order for the community to get on board, we should be super clear about this. My hope is that breaking changes are very minimal, and whatever breaking changes there are should be justified with a clearly argued rationale.

[jhamman]

To be honest, we're still sorting out the feasibility of maintaining API compatibility at the Array level. @d-v-b has been leading this so I'll let him share his thoughts.

[d-v-b]

There are a couple fundamental challenges for preserving the zarr-python v2 api after we fully integrate v3: The Array class in zarr-python exposes all the attributes defined for zarr version 2 arrays. But zarr version 3 arrays have different attributes, so in zarr-python version 3 we have a few options for dealing with this:

1. Uses the current zarr-python Array class to wrap v3 arrays and make them look like v2 arrays. this maximizes backwards compatibility, but would be a usability nightmare and confusing for people who want to work primarily with v3 arrays.
2. Create a new Array class in zarr-python that exposes zarr version 3 array attributes, much like zarr-python v2 did for zarr version 2 arrays, and then use this class to wrap zarr version 2 arrays. This would also be unpleasant, because zarr version 3 arrays don't have a dtype property (instead, that attribute is called data_type), and so hewing too close the zarr version 3 array spec for zarr-python would result in an impedance mismatch with the wider pydata ecosystem.
3. elevate the Array class to a level of abstraction above zarr version 2 and version 3 array specifications, and make that Array class capable of using zarr version 2 or 3 arrays as load-bearing members. This lets us optimize the Array class in zarr-python for a specific goal: a performant, familiar API for people coming from other pydata tools, while smoothing over the differences between v2 and v3 arrays. I think this is the best option, but it does involve substantial breaking changes to the zarr-python array API.

The only option here that doesn't break the zarr-python array API is 1, but it's a terrible option for people who want to get the most out of zarr version 3. The fact is, zarr v3 is a breaking change w.r.t. zarr v2, and I would suggest that people who do not want their code to break should continue using zarr v2 and pin zarr-python to version 2.

And this doesn't touch performance at all. I think we will need to change a lot of APIs to get good performance in zarr-python version 3.

[martindurant]

Do we need to sort out the ZEPs, agree to adopt them?

[joshmoore]

It's a good question but might be outside the scope of this document (since it's unsettled anyway).

[jhamman]

We've been proceeding assuming that some of the extensions that motivated V3 (sharding, variable chunk grids) would eventually find support here. But we aren't requiring full support for these extensions for the initial release.

[normanrz]

Fwiw, the sharding codec is already in the v3 branch. So, it could ship in the initial release.

[martindurant]

I feel like this still misses the point made above about concurrency of metadata operations and cross-variable operations. A sync Array object still needs to get its metadata and can only provide internal async IO.

We probably also need some concept of "batch" and/or "queue" to allow files to accumulate or stream to the async IO.

May be interesting to look into dask's batching mechanism for sending messages to the scheduler, which I don't personally have any knowledge of.

[d-v-b]

We probably also need some concept of "batch" and/or "queue" to allow files to accumulate or stream to the async IO.

I think tensorstore achieves something like this via transactions, it might be worth copying that API

[jhamman]

We have been discussing something similar that could exist at (or near) the store layer.

store = MyStore()

with store.transaction() as ts:
    array_foo.create(..., store=ts)
    array_bar.create(..., store=ts)

This is going to necessitate some thought around the state of the store before the transaction but would enable matching multiple writes in the form of a write-cache.

[normanrz]

Wouldn't it be possible for users to use the AsyncArray and AsyncGroup directly, if concurrency is important to them?

[martindurant]

You mean bundle the properties like shape?
The V2 version lists all the files to see if they exist or not, which we may well choose not to want.

[jhamman]

Agreed that listing the store is not a great default behavior. I'll add a note here.

[martindurant]

Mention cross-variable indexing? Like, the [0,0] value for each variable in a group?

[martindurant]

This is a big point! So here is the batching that I mentioned before. It should be clarified, what the state of store is before save/save_many is called: is it usable, does it have some ephemeral storage (including data?) ?

[jhamman]

Good questions @martindurant - I've left a note that indicates this part of the doc is in need of some additional design effort.

[martindurant]

(noting that fsspec has no dependencies of its own, but many of its implementations do)

[martindurant]

Part of the problem is not having a clear definition of which part of v2 is considered to outward-facing API. I would suggest it's Group and Array only (plus a few convenience functions like open_*).

[joshmoore]

rabernat 2 weeks ago
I don't think this needs to be in the docs itself. Better as it is, as a standalone doc for devs.

Several initial thoughts at a high-structural level on the design document:

• Agreed that it needs not be in the docs.
• Though I think we can make even the docs use .md
• I'd feel slightly better if this were down one directory level but that can only be the docs/ folder at the moment, so maybe we just punt on that.
• I could imagine getting this merged pretty proactively and then perhaps keep a modification list as suggestions are made. That might be easier on the v3 branch.
• What is the process you were envisioning, @jhamman? (As I ask that, I can't help but think of the ZEP process...)

[joshmoore]

It's a good question but might be outside the scope of this document (since it's unsettled anyway).

[jhamman]

I had intentionally not ordered but here is my own view on priorities here.

Suggested change

	- Provide a complete implementation of Zarr V3 through the Zarr-Python API
	- Align the Zarr-Python array API with the [array API Standard](https://data-apis.org/array-api/latest/)
	- Clear the way for exciting extensions / ZEPs (i.e. [sharding](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html), [variable chunking](https://zarr.dev/zeps/draft/ZEP0003.html), etc.)
	- Provide a developer API that can be used to implement and register V3 extensions
	- Improve the performance of Zarr-Python by streamlining the interface between the Store layer and higher level APIs (e.g. Groups and Arrays)
	- Clean up the internal and user facing APIs
	- Improve code quality and robustness (e.g. achieve 100% type hint coverage)
	- Provide a complete implementation of Zarr V3 through the Zarr-Python API
	- Clear the way for exciting extensions / ZEPs (i.e. [sharding](https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html), [variable chunking](https://zarr.dev/zeps/draft/ZEP0003.html), etc.)
	- Provide a developer API that can be used to implement and register V3 extensions
	- Improve the performance of Zarr-Python by streamlining the interface between the Store layer and higher level APIs (e.g. Groups and Arrays)
	- Clean up the internal and user facing APIs
	- Improve code quality and robustness (e.g. achieve 100% type hint coverage)
	- Align the Zarr-Python array API with the [array API Standard](https://data-apis.org/array-api/latest/)

[joshmoore]

Since the v3 branch doesn't have a "landing page", I'm going to misuse this PR for a moment: @jhamman, @normanrz, @d-v-b-- thoughts on opening a PR from v3 to main so everyone is aware of the full diff and has a place to comment?

[jhamman]

Per the discussion in the refactor meeting today, we're hoping to merge this and start in its current state and do additional design iteration in smaller PRs.

[jhamman]

@zarr-developers/python-core-devs - we need an approval to merge this.

[joshmoore]

joshmoore commented last month
Since the v3 branch doesn't have a "landing page", I'm going to misuse this PR for a moment: @jhamman, @normanrz, @d-v-b-- thoughts on opening a PR from v3 to main so everyone is aware of the full diff and has a place to comment?

Big 👍 for the merging of the doc. Any thoughts on the timeline for opening the PR for review?

[jhamman]

Big 👍 for the merging of the doc. Any thoughts on the timeline for opening the PR for review?

I think we're waiting on a few of the major reorg prs to land: #1589, #1590, #1634, and #1634. Once those are in, a review branch will probably make more sense.

[joshmoore]

Awesome. Thanks, @jhamman!