Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate non-future parallelism? #441

Closed
3 tasks
kendonB opened this issue Jun 29, 2018 · 2 comments
Closed
3 tasks

Deprecate non-future parallelism? #441

kendonB opened this issue Jun 29, 2018 · 2 comments

Comments

@kendonB
Copy link
Contributor

kendonB commented Jun 29, 2018

There is duplication of ideas and effort in the multiple forms of parallelism supported by drake/drake-via-future, adding to (at least my) confusion over how to best use the package. I think it would be best to just use future for parallelism then have a different argument for transient/persistent/staged.

This requires (at least):

  • Adding support for staged parallelism using future.
  • Adding a future evaluator for using Makefile parallelism. i.e. future::plan(makefile) OR deprecating Makefile parallelism.
  • Adding deprecation messages in the relevant functions

Also mentioned here but didn't get it's own issue: #227

@wlandau
Copy link
Member

wlandau commented Jun 30, 2018

It's a good thought, but I do not think it is time to deprecate any parallel drake is still in a brainstorming phase when it comes to HPC. New paths and technologies are emerging fast, and it is not yet clear where drake will settle. Also, in future-based parallelism specifically, there is extra overhead to make sure workers can be deployed remotely. This does not bode well for imports or large numbers of quick targets.

@wlandau wlandau closed this as completed Jun 30, 2018
@wlandau
Copy link
Member

wlandau commented Jun 30, 2018

Lastly, I have actually been trying to get rid of staged parallelism. I could not quite do it for the last CRAN release because there are some projects that benefit (e.g. #369 (comment)), and for projects that require remote workers, I believe transient workers ("future" parallelism) is uniformly better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants