Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelism on windows #123

Closed
dblodgett-usgs opened this issue Dec 20, 2024 · 0 comments
Closed

Parallelism on windows #123

dblodgett-usgs opened this issue Dec 20, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@dblodgett-usgs
Copy link
Collaborator

Windows does not support mcapply, the multi-core parallel approach that's available on Linux and Mac. Passing an integer in as a cluster option has turned into a lot of complexity to support because it doesn't work out of the box with Windows.

We adopted pbapply because it abstracts some of the complexity of directly supporting multiple OSs, but the more I've used it, the more I've just run into problematic edge cases.

There are two directions we could take this. One, which I've implemented in testing this stuff out, is to just take on the complexity but try to get things right and well tested. The other option would be to limit what the package supports for parallelism to just limit complexity entirely. That would be to either only support parallel clusters that are pre-defined or to only support future.

I'll open a new PR with the implementation I worked up as a point of discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant