Add LSF

[mrocklin]

It might be valuable to extend this repository with a solution for LSF. My hope is that this is relatively easy for someone with modest LSF experience. Looking at the current solutions for PBS or SLURM might be helpful (they're about 100 lines, mostly docstrings)

[jakirkham]

Happy to give this a go. We use LSF at work. So this should be pretty useful.

[lzamparo]

@jakirkham if it helps, I tried making a dask-distributed LSF script a while back. I encountered an error I couldn't fix, but maybe you can? Here's a gist.

[lesteve]

Just to split the task in smaller chunks:

• add a LSFCluster that inherits from dask_jobqueue.core.JobQueueCluster
• define LSFCluster.submit_command (probably bsub?) and LSFCluster.cancel_command (probably bkill?)
• implement LSFCluster._job_id_from_submit_output that takes a string (stdout output from the submit_cmd) and turns into into a job identifier.

Once implemented, it would be great to test it on a LSFCluster that you have at your disposal.

Looking at dask_jobqueue/slurm.py, dask_jobqueue/pbs.py, dask_jobqueue/sge.py is a good way to get started too.

[jakirkham]

Sorry have been busy with other things lately. If someone has interest and time, they should feel free to go ahead.

[lesteve]

If someone has interest and time, they should feel free to go ahead.

Yep, that was what I had in mind by specifying the steps in more details.

[lzamparo]

@lesteve I can give this a go; thanks for the direction. Is there a specific fixture-based test implemented for other cluster methods, or am I free to try some of my own tasks?

[lesteve]

It'd be great if you could give this a go! For the first iteration, I think you can try to put together and run a small snippet, e.g. something along these lines and get it to work on your local LSFCluster:

from dask_jobqueue import LSFCluster, Client

cluster = LSFCluster(...)  # use some arguments that make sense on your local cluster
client = Client(cluster)
result = client.map(lambda x: x + 1, range(10))
client.gather(result)

For tests, you could probably took some inspiration from the existing tests from test_slurm.py, test_pbs.py, etc ...
Personally I would be in favour of doing that in a separate PR.

[lesteve]

@lzamparo just so you know and to avoid duplicating work, there is an ongoing PR at #78. It would be great if you can give it a go (nowish or once the PR is merged, your call really) on your local LSF cluster and tell us whether that works for you!

[raybellwaves]

Still working through my PR and trying to get it to work. Subsequent testing (once working) on other LSF clusters will be great.
Leaving this link here: https://slurm.schedmd.com/rosetta.pdf which is helping me adapt the Slurm and PBS codes.

[lzamparo]

@lesteve Thanks for the heads up. Looks like @raybellwaves is making good headway, I'll step aside.

[lesteve]

@lzamparo I'll ping you when #78 is merged and it will be great if you could give it a go on your LSF cluster!

[raybellwaves]

While #78 is almost finished, i stumbled across https://github.com/IBMSpectrumComputing/lsf-python-api. May be of interest for the future e.g. implementing something like bjobs to check if the dask workers are running or pending. (#11)

[raybellwaves]

This has been merged now. Happy to know if it works on other LSF schedulers. Not just UM Pegasus.
One thing which cropped up for me again was the psutil issue at showed as my first comment in the PR. If this comes up for others we could add psutil to the dependencies as installing psutil made it go away. But If it's just something unique to pegasus I'll just have to remember to install it.

[lesteve]

I am going to close this one since the associated PR has been merged.

@raybellwaves note you can use "Fix #issueNumber" in your PR description, this way the associated issue gets closed automatically when the PR is merged. For more details, look at this.

About the psutil problem that you bumped into, I think it's quite hard to guess what the root cause is but I would bet that it was a problem with your environment (somehow you ended up with a broken psutil) rather than with with dask-jobqueue.

Note psutil is already in the dependencies because dask-jobqueue depends on distributed which depends on psutil.