Release tasks early

[hannahhoward]

Goals

Improve scaling by allowing workers to run in the background over parallelism limit for storage tasks once data transfer is finished

Implementation

Rather than n-workers queues in parallel, always kick off a seperate go routine per task.

We basically have two modes of popping tasks:

• track total tasks running
• below parallism limit, pop tasks and run them up to parallelism limit, using a existing delay if there are no tasks to pop
• above parallelism limit, pop no tasks until a task is released by a worker, either by the worker calling a release callback, or by the task finishing (also has nice side effect of not attempting a pop unless there is the ability to run it immediately)

Also:

• setup the whole engine so it's dependencies are mockable
• use a timer mock library (normal timer in production)
• setup tests to test the timing (these ended up being pretty complicated and I'm not totally sure the value of them... but...)