-
-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Libsubmit #182
Comments
Thanks for sharing @dgasmith, that is really interesting. And I mean that for both the Parsl project and the libsubmit part. When seeing #49, or also https://github.com/jupyterhub/batchspawner, that's a lot of libraries implementing the same kind of interfaces with jobqueue systems. Libsubmit is the first I see with a goal of extracting only the "submit anything" part, and could maybe be used here. Taking a quick look, it seems overkill and as you said, maybe too Parsl-specific yet, but having such a library would be a great help for many projects. And yes, at least we can benefit from the experience of dealing with other Jobqueue systems such as Condor, cobalt or others, so thanks again! Just out of curiosity, do you use Parsl? I can see it's not the same purpose as Dask, but there's some similarities, especially with Dask Delayed, did you compare the two softwares somehow? |
I do use Parsl and Dask, but just as a flat queue to ship jobs out to compute nodes rather than full blown workflows. So I cannot comment too much on the more specific features of the libraries. I agree that having a central place for spawning tech would be ideal. I have a close enough association with the Jupiter and Parsl folks that I could likely setup a conversation if desired. |
So why are you using both libraries? It's sound like Parsl would be more adapted to this use case?
Currently I don't see the need for that, but maybe in the future! Thanks. |
We have a database and workflows project for the quantum chemistry and treat Dask/Parsl/Balsam/Radical/BOINC/etc as general distributed computing libraries while rarely leveraging their workflow components. Users are free to pick the distributed computing project that they know and matches their use case. Something like Dask-jobqueue will allow Dask users to more easily access supercomputing resources that, granted, is similar to Parsl's functionality. |
Thanks for this information @dgasmith, closing for now. |
Searching for alternatives to wrap cluster schedulers I ran across libsubmit from the Parsl project. It may be too Parsl-specific for this project, but could provide nice templates for things like Cobalt/Jetstream/etc that are not yet here. I thought it would be good to at least be aware of this project.
The text was updated successfully, but these errors were encountered: