Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Learning from ipyparallel #49

Closed
jakirkham opened this issue May 1, 2018 · 4 comments
Closed

Learning from ipyparallel #49

jakirkham opened this issue May 1, 2018 · 4 comments

Comments

@jakirkham
Copy link
Member

ipyparallel (formerly part of the Jupyter Notebook) provides similar functionality to dask distributed and dask-jobqueue in that it allows users to startup a cluster and submit work to it. The model of the ipyparallel is a bit different from that of dask distributed, but that doesn't really concern us here.

What is interesting is ipyparallel has a trove of knowledge regarding starting jobs on various common HPC Schedulers. This knowledge is largely baked into one file. For HPC Schedulers already in dask-jobqueue, it's worth comparing notes to ipyparallel and see what can be learned on this front. As to the schedulers not present in dask-jobqueue, it's worth taking a look at ipyparallel's implementations and seeing what can be gleaned from it and how it might be used here. It's probably also worth learning how things have been refactored out in ipyparallel to see if there are any useful strategies for modeling HPC Schedulers generally.

@mrocklin
Copy link
Member

mrocklin commented May 1, 2018 via email

@jhamman
Copy link
Member

jhamman commented May 2, 2018

Thanks for sharing. I had a quick skim. There are some remarkable similarities between the two projects, particularly on the level of abstraction between different scheduling systems. May be worth using this as a starting point when developing the LSF cluster (#4).

@guillaumeeb
Copy link
Member

Wow, big source file! Scary at first, but a lot of information in there, probably worth taking a second look.

Thanks for sharing.

@guillaumeeb
Copy link
Member

Closing this for now, we may want to look at it again for some other scheduler implementation, was useful for LSF. We may also want to look at this for #133, along with https://github.com/jupyterhub/batchspawner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants