Add lsf #78

raybellwaves · 2018-06-27T04:44:45Z

working on #4

Thanks to @jhamman for his (in person) help with this. Hopefully I can get this squared away by the end of the week.

FYI I'm testing on python 3.6 at pegasus (University of Miami's HPC) and I was getting an un-obvious psutil error (see below). After installing the dependencies I installed a conda version of psutil (conda install -c conda-forge psutil) and it went away.

In [1]: from dask_jobqueue import LSFCluster
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-8ba34d16bb88> in <module>()
----> 1 from dask_jobqueue import LSFCluster

~/PYTHON3/dask-jobqueue/dask_jobqueue/__init__.py in <module>()
      1 # flake8: noqa
      2 from . import config
----> 3 from .core import JobQueueCluster
      4 from .moab import MoabCluster
      5 from .pbs import PBSCluster

~/PYTHON3/dask-jobqueue/dask_jobqueue/core.py in <module>()
      8 import dask
      9 import docrep
---> 10 from distributed import LocalCluster
     11 from distributed.deploy import Cluster
     12 from distributed.utils import get_ip_interface, ignoring, parse_bytes, tmpfile

~/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/distributed/__init__.py in <module>()
      3 from . import config
      4 from dask.config import config
----> 5 from .core import connect, rpc
      6 from .deploy import LocalCluster, Adaptive
      7 from .diagnostics import progress

~/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/distributed/core.py in <module>()
     23                    unparse_host_port, get_address_host_port)
     24 from .metrics import time
---> 25 from .system_monitor import SystemMonitor
     26 from .utils import (get_traceback, truncate_exception, ignoring, shutting_down,
     27                     PeriodicCallback, parse_timedelta)

~/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/distributed/system_monitor.py in <module>()
      2 
      3 from collections import deque
----> 4 import psutil
      5 
      6 from .compatibility import WINDOWS

~/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/psutil/__init__.py in <module>()
     97     PROCFS_PATH = "/proc"
     98 
---> 99     from . import _pslinux as _psplatform
    100 
    101     from ._pslinux import IOPRIO_CLASS_BE  # NOQA

~/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/psutil/_pslinux.py in <module>()
     24 from . import _common
     25 from . import _psposix
---> 26 from . import _psutil_linux as cext
     27 from . import _psutil_posix as cext_posix
     28 from ._common import ENCODING

ImportError: /nethome/rxb826/local/bin/miniconda3/envs/d-jq-test/lib/python3.6/site-packages/psutil/_psutil_linux.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __intel_sse4_strncpy

mrocklin · 2018-06-27T10:49:28Z

dask_jobqueue/jobqueue.yaml

+    extra: ""
+    env-extra: []
+    job-cpu: null
+    job-mem: null


I wonder, can we use the memory and threads * processes entries to remove the need for these entries? This is probably also a question for other dask-jobqueue maintainers, as I know that these appear in other configurations.

mrocklin · 2018-06-27T10:51:44Z

It would be useful to have a basic test, even if it only performs sanity-checks on the header. Here is an example for SLURM

dask-jobqueue/dask_jobqueue/tests/test_slurm.py

Lines 11 to 21 in 4cd24dc

    
           def test_header(): 
        
               with SLURMCluster(walltime='00:02:00', processes=4, threads=2, memory='7GB') as cluster: 
        
                   assert '#SBATCH' in cluster.job_header 
        
                   assert '#SBATCH -J dask-worker' in cluster.job_header 
        
                   assert '#SBATCH -n 1' in cluster.job_header 
        
                   assert '#SBATCH --cpus-per-task=8' in cluster.job_header 
        
                   assert '#SBATCH --mem=27G' in cluster.job_header 
        
                   assert '#SBATCH -t 00:02:00' in cluster.job_header 
        
                   assert '#SBATCH -p' not in cluster.job_header 
        
                   assert '#SBATCH -A' not in cluster.job_header

mrocklin · 2018-06-27T10:51:55Z

Thanks for working on this!

jakirkham · 2018-06-27T17:37:55Z

Could you please share how this should be tested? In particular, would be good to have a short script to try starting up Dask on LSF and running some simple computation with it.

mrocklin · 2018-06-27T17:40:28Z

I would expect the following to work for any job-queue cluster

from dask_jobqueue import LSFCluster
cluster = LSFCluster()

from dask.distributed import Client
client = Client(cluster)

assert client.submit(lambda x: x + 1, 10).result() == 11

raybellwaves · 2018-06-27T19:40:12Z

Hitting a wall here.
https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/slurm.py is probably closest to the lsf submit.
There is some code in slurm.py which explicitly sets n to 1 then sets ncpus using #SBATCH --cpus-per-task but there is no such command is LSF see here
https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/slurm.py#L88-L94
not sure if I need to set -n as 1 at the start?

When I run client.submit(lambda x: x + 1, 10).result() at the moment it just sits. Actually when I exit ipython I see KeyError: <Task '<lambda>-ab8232d42c76821e2cfa669075dd420b' no-worker>

Doing

from dask_jobqueue import LSFCluster
cluster = LSFCluster()
cluster.job_script()

gives
'#!/bin/bash\n\n#BSUB -J dask-worker\n#BSUB -e dask-worker.err\n#BSUB -o dask-worker.out\n#BSUB -W 00:30\n#BSUB -n 8\n#BSUB -M 30518\n\n\n\n/nethome/rxb826/local/bin/miniconda3/envs/d-jq-test/bin/python -m distributed.cli.dask_worker tcp://10.10.0.13:46285 --nthreads 2 --nprocs 4 --memory-limit 8GB --name dask-worker-2 --death-timeout 60\n'

If I manually copy this to a sumbit script (submit.sh) e.g.

#!/bin/bash

#BSUB -J dask-worker
#BSUB -e dask-worker.err
#BSUB -o dask-worker.out
#BSUB -W 00:30
#BSUB -n 8
#BSUB -M 30518

/nethome/rxb826/local/bin/miniconda3/envs/d-jq-test/bin/python -m distribute
d.cli.dask_worker tcp://10.10.0.14:48549 --nthreads 2 --nprocs 4 --memory-li
mit 8GB --name dask-worker-2 --death-timeout 60

and do bsub < submit.sh it runs e.g.

Job is submitted to <cpp> project.
Job <16633929> is submitted to default queue <general>.
$ bjobs
JOBID     USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
16633929  rxb826  RUN   general    login3      8*n248      *sk-worker Jun 28 01:58

but within python I seem to be struggling to get any workers.

guillaumeeb · 2018-06-28T10:55:18Z

There is some code in slurm.py which explicitly sets n to 1 then sets ncpus using #SBATCH --cpus-per-task but there is no such command is LSF see here

You probably want to use the following options to indicate the use of one node with several processes:

#BSUB -R "span[hosts=1]"
#BSUB -n 8

see https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_admin/span_string.html or https://www.hpc.dtu.dk/?page_id=1401

guillaumeeb

Thanks for your work here, hoping my comments will help you finish this!

guillaumeeb · 2018-06-28T11:03:18Z

dask_jobqueue/lsf.py

+        logger.debug("Job script: \n %s" % self.job_script())
+
+    def _job_id_from_submit_output(self, out):
+        return out.split('.')[0].strip()


Here you need to either do a parsing of the output string of the job, which seems to be something like:

Job is submitted to <cpp> project. Job <16633929> is submitted to default queue <general>.

But you more likely want to find the correct option for just outputing the job ID after the bsub command. Unfortunatly I did not find such an option after a few minutes Google search.

guillaumeeb · 2018-06-28T11:03:22Z

dask_jobqueue/lsf.py

+    """, 4)
+
+    # Override class variables
+    submit_command = 'bsub'


From what I understand from your last comment, the submit_command should be bsub < instead of just bsub.

@raybellwaves any response to this comment?

raybellwaves · 2018-06-28T19:21:19Z

Thanks for the comments @guillaumeeb!

I think LSF is unique compared to the others in terms of job submitting. bsub is the command but typing $bsub submit.sh does not spool submit.sh (some explanation in the examples at the bottom here http://www.glue.umd.edu/lsf-docs/man/bsub.html) you need to do bsub < submit.sh. To work around this i've used the solution in https://stackoverflow.com/questions/45134260/submitting-an-lsf-script-via-pythons-subprocess-popen-without-shell-true
I've therefore edited core.py and may have broke other schedulers.

It now works when I set off workers:

In [1]: from dask_jobqueue import LSFCluster

In [2]: cluster = LSFCluster(walltime='00:02', processes=2, threads=1, memory='4GB')

In [3]: workers = cluster.start_workers(2)
Job is submitted to <cpp> project.

Job is submitted to <cpp> project.


In [4]: from dask.distributed import Client

In [5]: client = Client(cluster)

In [6]: client.submit(lambda x: x + 1, 10).result()
Out[6]: 11

Although It still just hangs without setting off any workers:

In [1]: from dask_jobqueue import LSFCluster

In [2]: from dask.distributed import Client

In [3]: cluster = LSFCluster(walltime='00:02', processes=2, threads=1, memory='4GB')

In [4]: client = Client(cluster)

In [5]: client.submit(lambda x: x + 1, 10).result() # does nothing

raybellwaves · 2018-07-02T01:36:27Z

Well my hack with submit_command in lsf.py and the subprocess.Popen in core.py broke cluster.stop_workers(workers) so I need to think of a solution for bsub < without butchering core.py.

guillaumeeb · 2018-07-02T11:57:48Z

See how this is done in ipyparallel: https://github.com/ipython/ipyparallel/blob/6.1.1/ipyparallel/apps/launcher.py#L1397. Maybe you could try to use a similar syntax.

There is still the problem of shell=True keyword.

Maybe we should change the way core.py is written and have dedicated start and stop functions, that could be easily overridden if needed. Or just add a launch_command() function that the other one will call, and that just do the Popen part. @jhamman, @mrocklin, @lesteve any thoughts?

lesteve · 2018-07-03T13:14:36Z

Maybe we should change the way core.py is written and have dedicated start and stop functions, that could be easily overridden if needed. Or just add a launch_command() function that the other one will call, and that just do the Popen part. @jhamman, @mrocklin, @lesteve any thoughts?

Not sure what the best way is, but it looks like we need some special treatment for LSF indeed since it takes stdin and not the script name. Maybe something like this (I am guessing this is similar to your launch_command suggestion):

# in JobQueueCluster
def submit_job(self, script_filename):
    return self._call(shlex.split(self.submit_command) + [script_filename])

def start_workers(self, n=1):
    ...
    with self.job_file() as fn:
        out = self.submit_job(fn)
        ...

# in LSFCluster
def submit_job(self, filename):
    # note popen_kwargs needs to be added to _call so we can pass shell=True
    return self._call(shlex.split(self.submit_command) + ['<', script_filename],
        popen_kwargs={'shell': True})

raybellwaves · 2018-07-04T02:12:02Z

Thanks for the discussion on this. I'll wait until a decision is made. Unfortunately, LSF seems to be the black sheep and it looks as though it will have to be handled specifically. A launch_command may work that way I won't break cluster.stop_workers(workers) which expects the normal subprocess.Popen.

I can work on the Docker files in the meantime. I've not used docker before so if anyone can point me to some resources for setting up LSF in docker that would be appreciated (actually i'd prefer this to be a separate PR).

Lastly, i'll make the changes to reflect the latest PR. Thanks @mrocklin for your work on that.

mrocklin · 2018-07-04T02:35:41Z

Something like what @lesteve proposes seems reasonable to me.

@raybellwaves what do you think we should do?

raybellwaves · 2018-07-11T04:11:16Z

Thanks for the suggestion @lesteve. I believe i've got it working for myself and hopefully it should still work for others. I'll have to test tomorrow though as the queue is jammed.

lesteve · 2018-07-11T10:14:08Z

Great to hear, let us know when you think this is ready for review!

mrocklin · 2018-07-11T12:00:18Z

Thanks! Someone was asking me about LSF support just yesterday, so I know that this will be a welcome change!

…

On Wed, Jul 11, 2018 at 6:14 AM, Loïc Estève ***@***.***> wrote: Great to hear, let us know when you think this is ready for review! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszMnieHsq7rgnmrJQIatKbZRiz5j6ks5uFc_zgaJpZM4U5DCl> .

raybellwaves · 2018-07-11T21:24:29Z

Fixing the tests

mrocklin

Looking good! I'll be happy to see this in. A couple small comments.

mrocklin · 2018-07-11T22:19:42Z

dask_jobqueue/core.py

+            self.shell = True
+            return self._call(piped_cmd)
+        else:
+            return self._call(shlex.split(self.submit_command) + [script_filename])


Would it be possible to keep the core solution simple and instead put the LSF-specific implementation on the LSFCluster class?

Ah I see what @lesteve was suggesting

mrocklin · 2018-07-11T22:21:19Z

dask_jobqueue/lsf.py

+        if walltime is None:
+            walltime = dask.config.get('jobqueue.%s.walltime' % self.scheduler_name)
+        if job_extra is None:
+            job_extra = dask.config.get('jobqueue.lsf.job-extra')


Should this also use the self.scheduler_name pattern as above?

raybellwaves · 2018-07-17T19:23:16Z

@jhamman good spot. Everything is working again.
I'll spend a little time on suppressing the Job is submitted to <PROJECT> project. output before another review.

raybellwaves · 2018-07-18T02:22:36Z

The Job is submitted to <PROJECT> project. is the stderr of the bsub command. I redirected it to nowhere in my piped_cmd.

raybellwaves · 2018-07-26T20:52:32Z

Pinging @mrocklin @jhamman. Sorry to bother you. This is ready for review when you have time.

mrocklin

Sorry for the delay @raybellwaves . Generally this looks good. I've highlighted a couple of points that I think we can improve somewhat easily. Let me know what you think.

mrocklin · 2018-07-27T01:41:48Z

dask_jobqueue/core.py

@@ -280,13 +282,16 @@ def job_file(self):
                f.write(self.job_script())
            yield fn

+    def submit_job(self, script_filename):
+        return self._call(shlex.split(self.submit_command) + [script_filename])


This should probably be a private method so that users don't get the wrong idea that they should use it to submit jobs.

mrocklin · 2018-07-27T01:42:56Z

dask_jobqueue/jobqueue.yaml

+    env-extra: []
+    ncpus: null
+    mem: null
+    job-extra: []


Just checking in, does LSF need/use all of these options? I'm slightly concerned that as we copy configs from different systems we may accrue more than is necessary.

The only ones I haven't used are extra and env-extra. extra is for Additional arguments to pass to dask-worker so that is probably worth keeping. env-extra is other commands to the script before launching the worker. I can't see myself using this but core.py checks for it

dask-jobqueue/dask_jobqueue/core.py

Line 109 in f7c565a

env_extra : list

A future PR could be to move that out of core.py and have users specify it in their individual classes. I'll let you decide that.

mrocklin · 2018-07-27T01:43:09Z

dask_jobqueue/lsf.py

+    """, 4)
+
+    # Override class variables
+    submit_command = 'bsub'


@raybellwaves any response to this comment?

mrocklin · 2018-07-27T01:44:41Z

dask_jobqueue/lsf.py

+        `Job is submitted to <PROJECT> project.` which is the stderr and
+        `Job <JOBID> is submitted to (default) queue <QUEUE>.` which is the stdout.
+        Supress the stderr by redirecting it to nowhere.
+        The `piped_cmd` looks like ['bsub < tmp.sh 2> /dev/null'] """


This docstring looks a bit wonky. It looks more like a developer comment than a documentation string for users. If it is supposed to be a docstring then you might want to follow the numpydoc standard, or take a look at the dask developer notes on docstrings: http://dask.pydata.org/en/latest/develop.html#docstrings

mrocklin · 2018-07-27T01:45:15Z

dask_jobqueue/lsf.py

+        `Job <JOBID> is submitted to (default) queue <QUEUE>.` which is the stdout.
+        Supress the stderr by redirecting it to nowhere.
+        The `piped_cmd` looks like ['bsub < tmp.sh 2> /dev/null'] """
+        self.popen_shell = True


It seems odd to set this here again. It was already set above on the class, right?

mrocklin · 2018-07-27T01:47:29Z

dask_jobqueue/lsf.py

+        Supress the stderr by redirecting it to nowhere.
+        The `piped_cmd` looks like ['bsub < tmp.sh 2> /dev/null'] """
+        self.popen_shell = True
+        piped_cmd = [self.submit_command+' < '+script_filename+' 2> /dev/null']


There are some minor style issues. I recommend running flake8 on the codebase.

mrocklin@carbon:~/workspace/dask-jobqueue$ flake8 dask_jobqueue dask_jobqueue/lsf.py:5:1: F401 'os' imported but unused dask_jobqueue/lsf.py:69:13: F841 local variable 'memory' is assigned to but never used dask_jobqueue/lsf.py:124:41: E226 missing whitespace around arithmetic operator dask_jobqueue/lsf.py:124:47: E226 missing whitespace around arithmetic operator dask_jobqueue/lsf.py:124:63: E226 missing whitespace around arithmetic operator

I've opened #106 to discuss including this in the CI

mrocklin · 2018-07-27T01:51:43Z

dask_jobqueue/lsf.py

+
+def lsf_format_bytes_ceil(n):
+    """ Format bytes as text
+    LSF expects megabytes


@raybellwaves any response to this comment?

mrocklin · 2018-07-27T01:52:30Z

dask_jobqueue/lsf.py

+
+def lsf_format_bytes_ceil(n):
+    """ Format bytes as text
+    LSF expects megabytes


Also it would be good to format this docstring like http://dask.pydata.org/en/latest/develop.html#docstrings

mrocklin · 2018-07-27T01:53:59Z

dask_jobqueue/lsf.py

+
+    def stop_jobs(self, jobs):
+        """ set `self.popen_shell = False` """
+        self.popen_shell = False


Ah, I see, you're using the class state to sneak a parameter into the ._call method. I think it's probably better to pass extra keyword arguments into the ._call method directly and avoid the extra state. I'll write a comment on this in the core.py file.

mrocklin · 2018-07-27T01:56:16Z

dask_jobqueue/core.py

@@ -322,6 +327,7 @@ def _calls(self, cmds):
        for cmd in cmds:
            logger.debug(' '.join(cmd))
            procs.append(subprocess.Popen(cmd,
+                                          shell=self.popen_shell,


I recommend that we pass **kwargs, or at least shell= from the _call method into this function call and avoid the state entirely.

def _call(self, cmd, **kwargs): return self._calls([cmd], **kwargs) def _calls(self, cmds, **kwargs): ... procs.append(subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs))

Then we can call this like self._call(cmd, shell=True) and avoid mucking about with the popen_shell state.

mrocklin · 2018-07-27T19:07:14Z

Then lets drop them

…

On Fri, Jul 27, 2018 at 11:23 AM, Ray Bell ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In dask_jobqueue/jobqueue.yaml <#78 (comment)>: > + memory: null # Total amount of memory per job + processes: 1 # Number of Python processes per job + + interface: null # Network interface to use like eth0 or ib0 + death-timeout: 60 # Number of seconds to wait if a worker can not find a scheduler + local-directory: null # Location of fast local storage like /scratch or $TMPDIR + + # LSF resource manager options + queue: null + project: null + walltime: '00:30' + extra: "" + env-extra: [] + ncpus: null + mem: null + job-extra: [] Good question. I'm not sure what extra and env-extra are for. I haven't been using them in my development/testing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszHUgEbl3e5IBRn_0qfsR_xpLdQ1tks5uK1qmgaJpZM4U5DCl> .

guillaumeeb

Just a comment on stop_jobs.

We are close, I believe that after stating about this one we will be able to merge, thanks again for the time taken.

guillaumeeb · 2018-07-31T14:53:07Z

dask_jobqueue/lsf.py

+        logger.debug("Stopping jobs: %s" % jobs)
+        if jobs:
+            jobs = list(jobs)
+            self._call([self.cancel_command] + list(set(jobs)), shell=False)


As there is no state anymore, and shell=False is default with popen, you probably don't need it here, and thus you probably don't need to redefine stop_jobs in Lsf implementation.

Thanks. Good spot.

mrocklin · 2018-07-31T19:35:13Z

This now seems fine to me. @raybellwaves can you verify that things work well on your LSF cluster?

raybellwaves · 2018-07-31T20:47:01Z

raybellwaves · 2018-07-31T20:48:01Z

We can also get feedback from folks in #4 after merging

mrocklin · 2018-07-31T22:13:35Z

Merging this tomorrow if there are no further comments. If anyone gets to this before I do and are ok with things I encourage you to merge if you think it's ready.

jhamman · 2018-07-31T22:34:52Z

Thanks @raybellwaves for sticking with this. The only thing I see this needing is a few docs.

See the {index,examples,configurations,api}.rst files for some good places to document the LSFCluster. Bare minimum would be api.rst.

mrocklin · 2018-07-31T22:41:06Z

Looks like the api.rst change would be a single line. I'm in favor of that :) It'd be nice to get this in.

…

On Tue, Jul 31, 2018 at 3:34 PM, Joe Hamman ***@***.***> wrote: Thanks @raybellwaves <https://github.com/raybellwaves> for sticking with this. The only thing I see this needing is a few docs. See the {index,examples,configurations,api}.rst files for some good places to document the LSFCluster. Bare minimum would be api.rst. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszPCGMi-TaCDRYzidozFQhR5_y4N9ks5uMNuNgaJpZM4U5DCl> .

guillaumeeb

Thanks for all that, I think this is ready.

mrocklin · 2018-08-01T12:31:14Z

This is in. Thank you @raybellwaves for implementing this! I think that it will be very valuable

lesteve · 2018-08-01T13:53:45Z

Very nice!

raybellwaves · 2018-08-02T02:23:07Z

Big thanks to your all for creating the package and your teachings.

mrocklin reviewed Jun 27, 2018

View reviewed changes

lesteve mentioned this pull request Jun 27, 2018

Add LSF #4

Closed

guillaumeeb reviewed Jun 28, 2018

View reviewed changes

rebase

caf89e2

raybellwaves force-pushed the add-lsf branch from 7e673a4 to caf89e2 Compare July 11, 2018 03:57

raybellwaves added 3 commits July 11, 2018 00:04

update local_cluster

9a5e5c8

update thread to cores

1e47f4d

add comma

b6d248a

raybellwaves changed the title ~~WIP: Add lsf~~ Add lsf Jul 11, 2018

raybellwaves added 3 commits July 11, 2018 17:44

typos

7fbd76a

one more typo

72f195c

add cores and memory parameters to config test

f9970b3

mrocklin reviewed Jul 11, 2018

View reviewed changes

move submit_job to lsf.py

702b1ec

supress Job is submitted to <PROJECT> project.

c1f85fd

suppress Job is submitted to <PROJECT> project in bsub command

8b872b3

mrocklin reviewed Jul 27, 2018

View reviewed changes

Cealn up. Use **kwargs in _call

fc29249

raybellwaves mentioned this pull request Jul 27, 2018

Drop extra and env-extra requirements in core.py #108

Closed

jhamman mentioned this pull request Jul 27, 2018

Add flake8 to tests or black to CI #106

Closed

guillaumeeb reviewed Jul 31, 2018

View reviewed changes

rm stop_jobs for lsy.py

ccc61cb

updates docs with lsf

22cbb60

guillaumeeb approved these changes Aug 1, 2018

View reviewed changes

guillaumeeb mentioned this pull request Aug 1, 2018

Release? #87

Closed

mrocklin merged commit 2319b22 into dask:master Aug 1, 2018

adamhaber mentioned this pull request Aug 12, 2018

LSFCluster won't start workers #123

Closed

guillaumeeb mentioned this pull request Oct 8, 2018

LSFCluster worker doesn't execute all threads on the same node #172

Closed

raybellwaves mentioned this pull request May 13, 2019

include xr-ed versions of properscorings crps_gaussian, crps_ensemble xarray-contrib/xskillscore#10

Merged

lesteve mentioned this pull request Aug 26, 2019

LSFCluster may be overly specific? #328

Closed

raybellwaves mentioned this pull request Apr 2, 2020

community: scaling xskillscore #404

Closed

Add lsf #78

Add lsf #78

Conversation

raybellwaves commented Jun 27, 2018

Choose a reason for hiding this comment

mrocklin commented Jun 27, 2018

mrocklin commented Jun 27, 2018

jakirkham commented Jun 27, 2018

mrocklin commented Jun 27, 2018

raybellwaves commented Jun 27, 2018 • edited Loading

guillaumeeb commented Jun 28, 2018

guillaumeeb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raybellwaves commented Jun 28, 2018 • edited Loading

raybellwaves commented Jul 2, 2018

guillaumeeb commented Jul 2, 2018

lesteve commented Jul 3, 2018 • edited Loading

raybellwaves commented Jul 4, 2018 • edited Loading

mrocklin commented Jul 4, 2018

raybellwaves commented Jul 11, 2018

lesteve commented Jul 11, 2018

mrocklin commented Jul 11, 2018 via email

raybellwaves commented Jul 11, 2018 • edited Loading

mrocklin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raybellwaves commented Jul 17, 2018

raybellwaves commented Jul 18, 2018 • edited Loading

raybellwaves commented Jul 26, 2018

mrocklin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raybellwaves Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrocklin commented Jul 27, 2018 via email

guillaumeeb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrocklin commented Jul 31, 2018

raybellwaves commented Jul 31, 2018

raybellwaves commented Jul 31, 2018

mrocklin commented Jul 31, 2018

jhamman commented Jul 31, 2018

mrocklin commented Jul 31, 2018 via email

guillaumeeb left a comment

Choose a reason for hiding this comment

mrocklin commented Aug 1, 2018

lesteve commented Aug 1, 2018

raybellwaves commented Aug 2, 2018

raybellwaves commented Jun 27, 2018 •

edited

Loading

raybellwaves commented Jun 28, 2018 •

edited

Loading

lesteve commented Jul 3, 2018 •

edited

Loading

raybellwaves commented Jul 4, 2018 •

edited

Loading

raybellwaves commented Jul 11, 2018 •

edited

Loading

raybellwaves commented Jul 18, 2018 •

edited

Loading

raybellwaves Jul 27, 2018 •

edited

Loading