-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consolidate concat_dim earlier in prepare_target #229
Conversation
Closes pangeo-forge#227 (maybe, might be worth testing)
# https://github.com/pangeo-forge/pangeo-forge-recipes/issues/214 | ||
# intersect the dims from the array metadata with the Zarr group | ||
# to handle coordinateless dimensions. | ||
def _consolidate_dimension_coordinate( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The diff here looks strange.
All I did was
- move the
if config.consolidate_dimension_coordinates
to its own function so that I could call it from two places (here and earlier inprepare_target
forconcat_dim
) - Let the caller pass in the
dims
to check (so that we can do justconcat_dim
early on, and the other dims later; I think this is what we want). - Added a check that
arr.chunks != arr.shape (to avoid rewriting
concat_dimin
finalize_target`, since it would already be consolidated)
@TomAugspurger I've been locked up on other things today but I'll try to run some testing with this tomorrow 👍 |
/run-test-tutorials |
As discovered by @sharkinsspatial, this implementation buggy (sorry). When we call |
@TomAugspurger - over in #236, I have removed the pre-initialization of the concat dim with 0 values. It doesn't seem to break anything (including the tutorial notebooks). If that is the main bottleneck, I would prefer that (simpler) solution over this (more complex) one. I think the real fix for #227 is upstream in fsspec, related by supporting |
p.s. If you plan to keep working on this branch, please rebase as there have been some big improvements to the test suite. |
Closing as stale/out-of-date. Thanks all! |
Closes #227
(maybe, might be worth testing).
A few notes: