Use unbound regridding method for dask wrapping #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
huard
approved these changes
Oct 16, 2020
|
@andersy005 I'm not familiar with this aspect of dask. Could you take a quick look and confirm this is reasonable? |
andersy005
approved these changes
Oct 20, 2020
There was a problem hiding this comment.
Overall, this looks great to me 👍
There are currently no tests in the test suite that use distributed dask client, so all I can say is that solved the problem on my machine with dask 2.17.
I'm okay with merging this PR as is for the time being. We should add some dask tests to the test suites to make sure that things are working properly. I'll work on this some time this week.
|
Thanks, will open new issue for dask testing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes JiaweiZhuang/xESMF#71 and #38.
There are currently no tests in the test suite that use distributed dask client, so all I can say is that solved the problem on my machine with dask 2.17.
From what I understand the problem of using a distributed client is that it must serialize everything it sends to its workers. In the
Regridderobject, we keep the ESMpy Grid/Mesh/Locstream objects, which are not serializable by dask. Thus the error and the workaround that dumped overwrote_grid_inand_grid_outtoNone(see JiaweiZhuang/xESMF#71). I figured out that those two objects were passed to dask only because we were wrappingRegridder.regrid_numpywhich is a bound method, meaning the wholeRegridderinstance has to passed together with the function to the workers.This PR creates the
Regridder._regrid_arraystatic method and passes the needed arguments through a dictionary that removes any internal reference to the instance.