Integration tests: GIL

[fjetter]

• What happens if workers frequently run functions that hold the GIL
• Are our default configurations configured robust to survive 1s, 5s, 30s, 5min GIL holding?
• Is Coiled relying on the GIL to be released at least every Xs?
• What happens if the worker repeatedly locks up?
• Can we define sane default configurations to make everything a bit more robust? [n.d. @crusaderky : this is follow-up remediation work]

xref dask/distributed#6282

[crusaderky]

In scope

Push the envelope until everything breaks.
Document where it breaks.
Understand if it's a reasonable limitation or if we want to pen in remediation work.

Two use cases:

• a sync task does not release the GIL because it's stuck in C space
• a sync task does not release the GIL because it's running pure Python bytecode

Out of scope

• Remediation work
• async tasks that do not yield
• WorkerPlugin's that do not yield
• Slow SchedulerPlugin's or scheduler extensions
• slow pickle/unpickle (see Integration tests: Slow pickle/unpickle #205)

[gjoseph92]

The graph we're running should involve worker<->worker communication. If it's just client.map(gil_func, range(100_000), I wouldn't expect as much to happen. The problems arise especially when the worker event loop has many other things to do while the GIL is also held.