-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix 'to_masked_array' computing dask arrays twice #3006
Conversation
In my own tests I've been using the following custom scheduler with
Does something like this exist in the xarray tests? If not, I could add it then add a dask test to the DataArray tests. |
I'd be happy to merge this fix. I think it's a vestige of when we used to always cached computations. I'd also love to have more comprehensive test coverage, and I like the look of your custom scheduler. But I'm not sure it's worth adding that many lines of test logic for what is essentially a one line fix. |
@shoyer Makes sense. Any idea what's up with the travis test? It doesn't look like it is from my changes. |
@djhoese It looks like something broke in the development version of dask. But that shouldn't hold up merging your fix here... |
* master: (31 commits) Add quantile method to GroupBy (pydata#2828) rolling_exp (nee ewm) (pydata#2650) Ensure explicitly indexed arrays are preserved (pydata#3027) add back dask-dev tests (pydata#3025) ENH: keepdims=True for xarray reductions (pydata#3033) Revert cmap fix (pydata#3038) Add "errors" keyword argument to drop() and drop_dims() (pydata#2994) (pydata#3028) More consistency checks (pydata#2859) Check types in travis (pydata#3024) Update issue templates (pydata#3019) Add pytest markers to avoid warnings (pydata#3023) Feature/merge errormsg (pydata#2971) More support for missing_value. (pydata#2973) Use flake8 rather than pycodestyle (pydata#3010) Pandas labels deprecation (pydata#3016) Pytest capture uses match, not message (pydata#3011) dask-dev tests to allowed failures in travis (pydata#3014) Fix 'to_masked_array' computing dask arrays twice (pydata#3006) str accessor (pydata#2991) fix safe_cast_to_index (pydata#3001) ...
@shoyer Any idea when there might be another release of xarray where this fix will be included? I'm teaching a tutorial at SciPy this year that is effected by this bug. Learners are starting to prepare for the tutorials and I'd like if they could have this fix before the day of the tutorial. |
hopefully out this week! I am going to wait another day or two for someone else to look at #3040 and then will probably be issuing a release. |
I ran in to this issue when using
imshow
on a dask-basedDataArray
and dask'sProgressBar()
context manager. I noticed that when usingmy_data_arr.plot.imshow
the dask array was being computed twice. This PR fixes that.Suggestions for tests and documenting this fix are welcome.
whats-new.rst
for all changes andapi.rst
for new API