Skip to content

variable-sized chunks with zarr v3 #10880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

variable-sized chunks with zarr v3 #10880

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9e54553
allow variable chunk sizes for zarr v3
keewis Oct 27, 2025
ebfe0be
change the error message to mention zarr v2
keewis Oct 27, 2025
e2fd4c3
infer the chunk shape(s) from the zarr v3 metadata
keewis Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions xarray/backends/zarr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ async def async_getitem(self, key):
)


def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name):
def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name, zarr_format):
"""
Given encoding chunks (possibly None or []) and variable chunks
(possibly None or []).
Expand All @@ -301,15 +301,18 @@ def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name):
# while dask chunks can be variable sized
# https://dask.pydata.org/en/latest/array-design.html#chunks
if var_chunks and not enc_chunks:
if zarr_format == 3:
Copy link
Collaborator Author

@keewis keewis Oct 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this check is probably not sufficient

return tuple(var_chunks)

if any(len(set(chunks[:-1])) > 1 for chunks in var_chunks):
raise ValueError(
"Zarr requires uniform chunk sizes except for final chunk. "
"Zarr v2 requires uniform chunk sizes except for final chunk. "
f"Variable named {name!r} has incompatible dask chunks: {var_chunks!r}. "
"Consider rechunking using `chunk()`."
)
if any((chunks[0] < chunks[-1]) for chunks in var_chunks):
raise ValueError(
"Final chunk of Zarr array must be the same size or smaller "
"Final chunk of a Zarr v2 array must be the same size or smaller "
Comment on lines 307 to +315
Copy link
Member

@TomNicholas TomNicholas Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct - it's unfortunately not as simple as "Zarr V3 supports variable-length chunking but Zarr V2 doesn't".

f"than the first. Variable named {name!r} has incompatible Dask chunks {var_chunks!r}."
"Consider either rechunking using `chunk()` or instead deleting "
"or modifying `encoding['chunks']`."
Expand Down Expand Up @@ -475,6 +478,7 @@ def extract_zarr_variable_encoding(
var_chunks=variable.chunks,
ndim=variable.ndim,
name=name,
zarr_format=zarr_format,
)
if _zarr_v3() and chunks is None:
chunks = "auto"
Expand Down Expand Up @@ -853,9 +857,18 @@ def open_store_variable(self, name):
)
attributes = dict(attributes)

if hasattr(zarr_array, "metadata"):
chunk_grid = zarr_array.metadata.chunk_grid
chunks = getattr(chunk_grid, "chunk_shapes", None)
# regular chunk grid
if chunks is None:
chunks = chunk_grid.chunk_shape
else:
chunks = zarr_array.chunks

encoding = {
"chunks": zarr_array.chunks,
"preferred_chunks": dict(zip(dimensions, zarr_array.chunks, strict=True)),
"chunks": chunks,
"preferred_chunks": dict(zip(dimensions, chunks, strict=True)),
}

if _zarr_v3():
Expand Down
Loading