-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
@distributed
fails on loops over iterators
#57
Comments
@distributed
fails on loops over iterators@distribute
fails on loops over iterators
@distribute
fails on loops over iterators@distributed
fails on loops over iterators
Does any one tell me whether the macro "@distributed" work on iterators now? I have a very large data array (more than 900,000,000) after combinations, collect() function will exhaust all memory. |
Yes, Note that distributing such a large array to your workers may swamp the computation time with communication overhead. You may want to consider "peeling off" one of your combinatorial elements, distribute that amongst your workers, and then have each of your workers compute the combinations of that one element with the remainder. |
Thanks, mbauman. I will try your suggestions. |
Hello Matt, if I understand correctly, this indexing requirement is the reason behind |
Yes, that's correct. Just as a breadcrumb, with JuliaLang/julia#32310 we could possibly add indexing support. |
Distributed
for
loops over iterators fail, seemingly because@distributed
expects the iterator to implementgetindex
. It makes sense that@distribute
needsgetindex
, but I don't see that in the docs. The possible solutions seem to be:@distributed
to mention this requirement.@distributed
to work for iterators.We could do JuliaLang/julia#1 until JuliaLang/julia#2 is finished.
Here's an example:
This is surprising, since the following non-parallel reduction works:
Adding a
collect
to the distributed version fixes it:Output:
600
The text was updated successfully, but these errors were encountered: