test_spilling flaky

[ian-r-rose]

tests/stability/test_spill.py::test_spilling, introduced in #229, has been quite flaky, failing in ~25% of it's test runs (example). Since it is tested across a wide test matrix, this means that most workflows fail on test_spill.

For the most part, it seems that they are failing in the client setup fixture during the wait_for_workers() call. It seems that the high memory usage from that test is making cluster restarts pretty unreliable.

cc @hendrikmakait

[hendrikmakait]

From what I have seen so far, those flakes seem to occur in #235, not main, is that correct?

[ian-r-rose]

Hmm, that's a good point. I'll dig a bit deeper

[hendrikmakait]

My guess is that the reused cluster fixture is either busy cleaning up after running the first spilling test or getting somehow corrupted.

[ian-r-rose]

My guess is that the cluster is either busy cleaning up after running the first spilling test or getting somehow corrupted.

I agree, I wonder if dask/distributed#6944 would help here as well

[ian-r-rose]

This test is sensitive to disk space, no? I wonder if package_sync uses more disk and changes the performance characteristics here.

[hendrikmakait]

It is; the disk size has been set through trial-and-error, I guess it won't hurt to just up it by another couple of GiBs.

[ian-r-rose]

I believe that the platform team recommends using at least a t3.large for package_sync. I'd imagine if we did this we'd want to change the size of the allocated data, correct?

[hendrikmakait]

Yes and no, by increasing the data size we'd stick with the original "goal" of writing 10x the memory size to disk but I'm wondering if we would lose a lot if we just stuck to the same size or something that only accounts for the additional 4 GiB of memory per worker. After all, writing to disk is so slow.

[ian-r-rose]

I'm trying a t3.large over here. If that improves things, would you have any objections, or would that harm the integrity of the test? :)