Mitigate stragglers

[tomwhite]

When running on Lithops, it's common to see a small number of tasks take a lot longer than the others. To mitigate this, we could have a rule like "if any tasks are taking more than x times as long as the first y% completed tasks then launch speculative duplicates for them". (E.g. x=2, and y=75.) This only works since tasks are idempotent.

[tomwhite]

There is quite a lot of literature on mitigating stragglers in distributed systems (e.g. https://www.sciencedirect.com/science/article/pii/S1319157822000659, https://www.usenix.org/legacy/event/osdi08/tech/full_papers/zaharia/zaharia_html/index.html). A lot of it however assumes that tasks are heterogeneous, or that tasks need to be optimally packed onto available workers, neither of which is the case here, so a simpler policy may suffice.

[tomwhite]

First attempt in 544fb62

[tomwhite]

We've got use_backups for Modal, Lithops and now Dask (#279), so I'm going to close this.