Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-stage rechunk #650

Open
tomwhite opened this issue Dec 17, 2024 · 0 comments
Open

Multi-stage rechunk #650

tomwhite opened this issue Dec 17, 2024 · 0 comments

Comments

@tomwhite
Copy link
Member

The rechunker algorithm has support for multi-stage rechunking, which reduces the number of IO operations.

It would be good to enable this in Cubed rechunk. The simplest way to do this would be to expose the min_mem parameter, but it might be interesting to see if there is a way of automatically finding the number of stages that optimizes for something we care about (e.g. min IO ops).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant