SLURM won't replace $JOB_ID placeholder

[ShigekiKarita]

Hi, I'm using SLURM 17.02.3.
In my environment, I noticed filenames of stdout and stderr are wrong (e.g., worker.$JOB_ID.err/out)

https://github.com/dask/dask-drmaa/blob/master/dask_drmaa/core.py#L42-L43

'outputPath': ':' + worker_out_path_template % dict(jid='$JOB_ID.$drmaa_incr_ph$', kind='out'),
'errorPath': ':' + worker_out_path_template % dict(jid='$JOB_ID.$drmaa_incr_ph$', kind='err'),

Solution

Instead of $JOB_ID, SLURM uses %j placeholder, please see https://slurm.schedmd.com/sbatch.html
jid='$JOB_ID.$drmaa_incr_ph$' should be jid='%j.$drmaa_incr_ph$' in slurm.
I confirmed that when I hard-cord it in slurm but I don't know how to determine slurm or any other grid engines.

[jakirkham]

Handling Job ID on LSF is a bit different than say SGE or SLURM ( #32 ). So also needs some hand holding to get the right filename.

[jakirkham]

Generally thinking we need some way to configure part of the filename for stdout or stderr. Starting with worker or better yet dask-worker. is fine and ending with .out or .err respectively is also fine. What would be good is to make the middle chunk (logfile_infix ?) configurable so one can include any %j, %J, or $JOB_ID or whatever else the user might want to add there (e.g. proj-v2.3.2). This should handle the different clusters using DRMAA with their conventions better while providing better functionality to the user. We can also leave the $JOB_ID part as default to avoid breaking existing use cases.

[jakirkham]

Trying to address this in PR ( #56 ). Feedback and help testing would be appreciated.