Close workers gracefully

[mrocklin]

This allows workers to optionally terminate themselves gracefully after a predetermined time. This can be helpful in a few contexts:

1. We receive a SIGINT, and know that we need to clean up quickly (though note that the signal handlers are not implemented as part of this PR (cc @jni))
2. We know that we'll be kicked off at a certain time, such as in one hour from now, as is often specified by HPC job schedulers (cc @lesteve @guillaumeeb @jhamman )
3. We just want to refresh our workers every once in a while, because we know that our code leaks some memory . Fixes Gracefully restart a worker from a nanny #2861

This is configurable as keywords to the Worker or Nanny classes, in config values, or with CLI. Here is an example with CLI.

Restart to clear state

dask-worker scheduler:8786 --lifetime 1hr --lifetime-restart --lifetime-stagger 5m

This will kill the worker roughly 1 hour from now +- a range of 5 minutes (to avoid killing all of our workers at the same time). It will also allow that worker to be restarted afterwards

Restart to avoid walltime death

dask-worker scheduler:8786 --lifetime 58m 

Here we don't try to restart the worker (no point) and we choose a time a bit before our 60m walltime.

Also cc @jacobtomlinson @jcrist for general deployment information

For folks who want to review. I recommend going commit-by-commit (it should be fairly clean). (this is also a decently simple/educational PR to review)

[mrocklin]

Merging this tomorrow if there are no further comments (Although comments would be quite welcome here)

[TomAugspurger]

though note that the signal handlers are not implemented as part of this PR

Do you anticipate this making fixing #2788 easier? Your comment in #2788 (comment) makes it sound like "yes".

[TomAugspurger]

Implementation looks good at a glance.

[mrocklin]

Do you anticipate this making fixing #2788 easier?

It's the action that I think we would want to take when handling a SIGINT.

[mrocklin]

Merging this shortly if there are no further comments

[guillaumeeb]

Thanks for this @mrocklin, just a fiew comment on the documentation.

[lesteve]

Just a quick comment (I don't think this is a problem for this PR, since the signal handling is only part of it AFAICT): for SGE the signal you get when the job exceed the walltime is a SIGKILL (e.g. kill -9). Not sure for other job schedulers.

[jacobtomlinson]

Ouch! AFAIK Slurm and PBS send a SIGTERM shortly before the SIGKILL to give the process a chance to exit cleanly.

[lesteve]

Good to know that other schedulers are a bit nicer regarding job termination.

I found this which seems to indicate that for SGE SIGKILL is used by default but that you can configure it to be something else.