Pass through dask-worker configuration options

[mrocklin]

The dask-worker process can be configured in a variety of ways:

• How many cores should it use?
• With what mix of threads and processes should it use those cores?
• How much RAM should it use? What fraction of available RAM should it occupy before spilling to disk?
• Should it serve HTTP diagnostics on a particular port
• etc..

We can pass these through fairly easily through the DRMAACluster.__init__ or start_workers methods somewhat trivially.

What is less trivial is if we want to control the number of CPUs or memory used by the DRMAA job and pair this with the dask-worker configuration. I'm somewhat concerned that DRMAA expects us to take up one CPU by default when we're impolitely taking up all of the CPUs on the host. It would be nice to either request a certain host configuration from the job scheduler or else observe what our expected budget is after our job is allocated.

[davidr]

I'll answer the last question first.

Like with #4 , this can also get a bit site-specific. By default, every scheduler I know of will only request one core by default. Sometimes that will be restricted to one actual core via cgroups or taskset and other times it will just be a gentleman's agreement that if you're requesting a core, you'll only use a core and nothing will actually prevent you from using all the cores on a box.

With Grid Engine and its derivatives, you request more than one core with a "parallel environment". The common ones are named, by convention (but by no means universally) "smp" and "mpi". These are also requested with template.nativeSpecification (nativeSpecification = "-pe smp 10" will for instance give you 10 cores on the same host. -pe mpi 10 will give you 10 cores, but not guaranteed to be on the same host and the process must be started with MPI. that's likely out of scope for this project. :)

Given that 1-core non-parallel-environment jobs will likely be scheduled fastest, the easiest thing to do is probably to schedule as many of those as possible in an array job (as DRMAACluster seems to be doing) and only use a single thread/core.

[mrocklin]

OK, using a single core by default seems like the sane default choice, perhaps coupled with some release valve to let people pass through their own specific options.

[jakirkham]

Have opened issue ( #66 ) to discuss more how resources could be configured by checking the scheduler implementation with DRMAA and thus passing through the appropriate flags with such settings. We can also forward this information to the dask-worker running in that job so that it is aware what the scheduler job was configured with.