sharding: lift constraint on evenly divisible subchunk shapes

[LDeakin]

Now that the rectilinear grid is moving forward, it is time to lift the constraint that the sharding_indexed subchunk shape must evenly divide the chunk shape.

The only question is: what should the shape of the straddling subchunks be?

• Bounded by the chunk shape
• Always matching the subchunk shape, with some fill value elements that get ignored

My vote is on the former, so we can save some bytes :).

I've evaluated this with zarrs and it was fairly straightforward to support in the codec itself.

xref: zarrs/zarrs#233

[d-v-b]

Bounded by chunk shape sounds good to me! I haven't looked into the zarr-python implementation (@jhamman is working on the rectilinear chunking PR) but I don't think it will be a big lift

[jbms]

I'd agree with bounding it by the outer chunk shape (meaning the sub-chunk grid is also rectilinear rather than regular).

[shoyer]

Now that the rectilinear grid is moving forward, it is time to lift the constraint that the sharding_indexed subchunk shape must evenly divide the chunk shape.

👍

The only question is: what should the shape of the straddling subchunks be?

• Bounded by the chunk shape
• Always matching the subchunk shape, with some fill value elements that get ignored

For clarity, could you give a example of what these would look like?

[jbms]

One thing is that when used with a regular grid, the original constraint, while not strictly necessary, is still likely to be a good idea in almost all cases so that the overall grid does not become non-regular accidentally. Implementations may want to "encourage" users to pick a sub-chunk shape that satisfies that constraint when using this codec with a regular grid, e.g. warn unless some other (not part of the stored metadata) option like allow_non_regular_subchunk_grid is specified by a user. Otherwise users may arbitrarily pick some chunk shapes and happen to end up with a non-regular grid.