Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Libsubmit #182

Closed
dgasmith opened this issue Oct 26, 2018 · 5 comments
Closed

Libsubmit #182

dgasmith opened this issue Oct 26, 2018 · 5 comments

Comments

@dgasmith
Copy link

dgasmith commented Oct 26, 2018

Searching for alternatives to wrap cluster schedulers I ran across libsubmit from the Parsl project. It may be too Parsl-specific for this project, but could provide nice templates for things like Cobalt/Jetstream/etc that are not yet here. I thought it would be good to at least be aware of this project.

@guillaumeeb
Copy link
Member

Thanks for sharing @dgasmith, that is really interesting. And I mean that for both the Parsl project and the libsubmit part.

When seeing #49, or also https://github.com/jupyterhub/batchspawner, that's a lot of libraries implementing the same kind of interfaces with jobqueue systems. Libsubmit is the first I see with a goal of extracting only the "submit anything" part, and could maybe be used here. Taking a quick look, it seems overkill and as you said, maybe too Parsl-specific yet, but having such a library would be a great help for many projects.

And yes, at least we can benefit from the experience of dealing with other Jobqueue systems such as Condor, cobalt or others, so thanks again!

Just out of curiosity, do you use Parsl? I can see it's not the same purpose as Dask, but there's some similarities, especially with Dask Delayed, did you compare the two softwares somehow?

@dgasmith
Copy link
Author

I do use Parsl and Dask, but just as a flat queue to ship jobs out to compute nodes rather than full blown workflows. So I cannot comment too much on the more specific features of the libraries.

I agree that having a central place for spawning tech would be ideal. I have a close enough association with the Jupiter and Parsl folks that I could likely setup a conversation if desired.

@guillaumeeb
Copy link
Member

I do use Parsl and Dask, but just as a flat queue to ship jobs out to compute nodes rather than full blown workflows

So why are you using both libraries? It's sound like Parsl would be more adapted to this use case?

I could likely setup a conversation if desired

Currently I don't see the need for that, but maybe in the future! Thanks.

@dgasmith
Copy link
Author

dgasmith commented Oct 30, 2018

We have a database and workflows project for the quantum chemistry and treat Dask/Parsl/Balsam/Radical/BOINC/etc as general distributed computing libraries while rarely leveraging their workflow components. Users are free to pick the distributed computing project that they know and matches their use case.

Something like Dask-jobqueue will allow Dask users to more easily access supercomputing resources that, granted, is similar to Parsl's functionality.

@guillaumeeb
Copy link
Member

Thanks for this information @dgasmith, closing for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants