Pool size limit

[hjoliver]

Pool size limit, with preferential release of older cycle points, as an alternative to cycle point based runahead limiting.

Experimental.

• currently has a hardwired limit.
• built on Revert to iterative finished task removal. #3861 (iterative task removal - to better see what's happen in the UIs)
• incorporates Hide waiting tasks from n=0. #3823 (keeps partial waiting tasks out of the main pool, to avoid causing a stall)

How it works:

• active tasks spawn on demand, into the runahead pool which is not size-limited
  • (under SoD all tasks in the runahead pool are ready to run once released)
• tasks in the runahead pool are released, earliest cycle points first, until the main pool reaches its size limit

This would not work under SoS, but under SoD it doesn't cause race-condition stalls because every task in the runahead pool is ready to run so it doesn't matter which of them get released to keep the flow running. The runahead pool is not size-limited - it takes all children that get spawned by the active task pool - but the smaller the active pool the fewer children they spawn and hence the smaller the runahead pool.

Results so far:

• successfully runs the old "complex" flow with a max pool size of 10 and very little load on the VM.

These changes partially address #xxx
These changes close #xxxx
This is a small change with no associated Issue.

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).

• Appropriate tests are included (unit and/or functional).
• Already covered by existing tests.
• Does not need tests (why?).

• Appropriate change log entry included.
• No change log entry required (why? e.g. invisible to users).

• (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
• (7.8.x branch) I have updated the documentation in this PR branch.
• No documentation update required.

• Created an issue at cylc-flow conda-forge repository with version changes (if you changed dependencies in setup.py, see recipe/meta.yaml).
• No dependency changes.

[hjoliver]

UPDATE - I've realized that in the new spawn-on-demand scheduler pool-size limiting does exactly the same thing (in effect) as setting a "default" (global) queue limit. So we can already do it. The only difference is that our queues release tasks on a FIFO basis and this branch releases tasks with older cycle points first. But either of those release strategies could be used with either method. So I'll probably close this PR as superseded by a new Issue tomorrow...

[hjoliver]

(Also worth noting that if you do set a queue limit then runahead limiting is really not needed, although both can be used at once).

[hjoliver]

(Leaving this up for now until I get time to consider cycle point priority queue release as an alternative to FIFO and runahead limit).

[hjoliver]

Closing as superseded by #3874 (plus a note in the spawn-on-demand proposal document)