Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handling arrays with non-uniform chunking #40

Open
jakirkham opened this issue Mar 16, 2018 · 3 comments
Open

Handling arrays with non-uniform chunking #40

jakirkham opened this issue Mar 16, 2018 · 3 comments
Labels
protocol-extension Protocol extension related issue

Comments

@jakirkham
Copy link
Member

Currently Zarr handles Arrays with uniform chunking. Meaning all chunks have the same size (excepting end chunks, which can be shorter). It would be nice if Zarr could also handle non-uniform chunking. Meaning that chunks would still live on a grid, but may vary in size based on their location (IOW not only end chunks would have this feature).

Motivating use cases include saving a Dask Array with non-uniform chunking. Admittedly these could be rechunked if the chunk size is known. Though this comes with some overhead compared to not rechunking. When the chunk size is unknown, this cannot be easily accomplished.
@jakirkham
Copy link
Member Author

This can be useful in two contexts: storing Dask Arrays as is ( zarr-developers/zarr-python#251 ) and handling shorter chunks on the ends ( zarr-developers/zarr-python#233 ).

@jakirkham
Copy link
Member Author

Should add, raised issue ( dask/dask#3302 ) upstream about rechunking Dask Arrays to be uniform.

@alimanfoo alimanfoo transferred this issue from zarr-developers/zarr-python Jun 20, 2019
This was referenced Feb 27, 2020
Open
Open
@jrbourbeau jrbourbeau mentioned this issue Mar 30, 2020
Closed
@rabernat rabernat mentioned this issue Sep 8, 2021
Open
@jakirkham jakirkham mentioned this issue Mar 23, 2022
Open
@jakirkham
Copy link
Member Author

cc @martindurant (who may have interest in this)

@martindurant martindurant mentioned this issue Oct 18, 2022
Merged
@jstriebel jstriebel added the protocol-extension Protocol extension related issue label Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
protocol-extension Protocol extension related issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jakirkham @jstriebel