engine: Regularly purge worker threads

[josephjclark]

The latest worker version will bypass node's module caching when importing adaptors.

This ensures an isolated environment, which is super great, but will result in a memory leak as old adaptors will not be unloaded/tidied/gc'd.

A good solution to this, which is also quite a nice security policy, is to regularly burn worker threads by calling workerpool.terminate().

We should:

• Set an interval to run every 10 minutes (?) (every hour?)
• Check whether there are any pending jobs
• Check whether any jobs have run since the last purge (maybe wait for at least N jobs to run before bothering to purge)
• If no pending jobs, and we have done work since the last purge, purge the worker threads.
• Emit an event to say we're purging (and maybe how many threads will die).

We can unit test on the emitted event.

Spinning this out into an issue because I don't think I can do this before I travel today.

[josephjclark]

Refinement #1: purge on complete

Refinement #2: maybe rate limit attempts, or purge before claim, or purge every 10 claims, or something

@stuartc is very worried that this is going to blow the stack quickly. Let's keep an eye on it, but I think we'll be looking at switching worker library very quickly.

[josephjclark]

Purge on complete is implemented, with logs and emitted events so we have some visibility.