Implement `cylc remove` proposal #6472

MetRonnie · 2024-11-11T11:58:04Z

Closes #5643. Supersedes #6370.

Summary

This fully implements the "Cylc Remove Extension" proposal.

Flow numbers

cylc remove now has a --flow option for removing a task from specific flows.

If not used, it will remove the task from all flows that it belongs to.

If the removed task is active/waiting, if it is removed from a subset of flows that it belongs to, it will remain in the task pool; if it is removed from all flows that it belongs to, it will be removed from the task pool (as is the current behaviour).

If a task is removed from all flows that it belongs to, it will become a no-flow task (flow=None).

For ease of reviewing, you can use my UI branch that displays flow numbers: https://github.com/MetRonnie/cylc-ui/tree/flow-nums ¹.

Historical tasks

cylc remove now can remove tasks that are no longer active, making it look like they never ran. It does this by removing the task from the specified flows in the database (in the task_states and task_outputs tables)², and un-setting any prerequisites of active tasks that the removed task had naturally satisfied³. If a task is removed from all flows that it belongs to, a no-flow task is left in the DB for provenance.

The above also applies to active/waiting tasks that cylc remove is used on.

Kill submitted/running tasks

Using cylc remove on a submitted/running task will now kill it if you are removing the task from all flows that it belongs to.

Unlike with cylc kill, downstream tasks will not spawn off the :fail or :submit-fail outputs as the task is in flow=none, and also the failed and submission failed handlers will not run.

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
No dependency changes
Tests are included
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at Update Major Changes Page cylc-doc#791
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

Waiting tasks that are not yet in the pool have greyed out flow numbers at the moment. ↩
If removing flows would result in two rows in the DB no longer being unique, the SQLite UPDATE OR REPLACE statement is used, so the first entry will be removed and the most recent entry will remain. ↩
Prerequisites manually satisfied by cylc set --pre are not affected by cylc remove. ↩

`json.dumps()`/`json.loads()` are relatively slow (~1us). But these functions are likely to be called many times with `flow={1}`.

…tables

…tasks in the given flow(s)

…fied prereqs

…am tasks

- Update data store with changed prereqs - Don't un-queue downstream task if: - the task is already preparing - the task exists in flows other than that being removed - the task's prereqs are still satisfied overall - Remove the downstream task from the pool if it no longer has any satisfied prerequisite tasks

…nums`

This will allow it to call the method to kill submitted/running tasks

hjoliver

Still part way through, looks great so far.

cylc/flow/task_state.py

cylc/flow/command_validation.py

cylc/flow/platforms.py

cylc/flow/scheduler.py

cylc/flow/scripts/remove.py

oliver-sanders · 2024-11-26T15:02:51Z

(Does that mean you now agree with the behavior on this branch?)

Ish, there are two schools of thought, but I'm happy to go ahead with this.

If downstream tasks get removed, I think that should be logged at INFO level rather than DEBUG. It's potentially quite important information.

Note: We are only talking about SoD pre-spawning here (as we only remove children with partially satisfied prereqs).

SoD pre-spawning is really an implementation detail, so we could argue that these tasks should not be logged (I completely understand why Ronnie has logged this at DEBUG level). However, on the other hand, the n-window is now part of the model, it defines what the GUI sees and what commands users can issue against the task, so it is pertinent information.

Given that we log tasks that are pre-spawned by SoD at the INFO level, I agree that we should log their removal at the INFO level too.

oliver-sanders · 2024-11-25T16:56:37Z

cylc/flow/scheduler.py

+        removed: Dict[str, FlowNums] = {}
+        not_removed: Set[str] = set()
+        to_kill: List[TaskProxy] = []
+
+        for itask in active:
+            fnums_to_remove = itask.match_flows(flow_nums)
+            if not fnums_to_remove:
+                not_removed.add(itask.identity)
+                continue
+            removed[itask.identity] = fnums_to_remove
+            if fnums_to_remove == itask.flow_nums:
+                # Need to remove the task from the pool.
+                # Spawn next occurrence of xtrigger sequential task (otherwise
+                # this would not happen after removing this occurrence):
+                self.pool.check_spawn_psx_task(itask)
+                self.pool.remove(itask, 'request')
+                to_kill.append(itask)
+                itask.removed = True
+            itask.flow_nums.difference_update(fnums_to_remove)
+
+        matched_task_ids = {
+            *removed.keys(),
+            *(quick_relative_id(cycle, task) for task, cycle in inactive),
+        }
+
+        for id_ in matched_task_ids:
+            point_str, name = id_.split('/', 1)


It is not necessary to form these cycle/task components into IDs only to split them back into components, we can just store them as tuples for later access.

(untested)

Suggested change

removed: Dict[str, FlowNums] = {}

not_removed: Set[str] = set()

to_kill: List[TaskProxy] = []

for itask in active:

fnums_to_remove = itask.match_flows(flow_nums)

if not fnums_to_remove:

not_removed.add(itask.identity)

continue

removed[itask.identity] = fnums_to_remove

if fnums_to_remove == itask.flow_nums:

# Need to remove the task from the pool.

# Spawn next occurrence of xtrigger sequential task (otherwise

# this would not happen after removing this occurrence):

self.pool.check_spawn_psx_task(itask)

self.pool.remove(itask, 'request')

to_kill.append(itask)

itask.removed = True

itask.flow_nums.difference_update(fnums_to_remove)

matched_task_ids = {

*removed.keys(),

*(quick_relative_id(cycle, task) for task, cycle in inactive),

}

for id_ in matched_task_ids:

point_str, name = id_.split('/', 1)

removed: Dict[Tuple(str, str), FlowNums] = {}

not_removed: Set[Tuple(str, str)] = set()

to_kill: List[TaskProxy] = []

for itask in active:

key = (itask.tokens.cycle, itask.tokens.task)

fnums_to_remove = itask.match_flows(flow_nums)

if not fnums_to_remove:

not_removed.add(key)

continue

removed[key] = fnums_to_remove

if fnums_to_remove == itask.flow_nums:

# Need to remove the task from the pool.

# Spawn next occurrence of xtrigger sequential task (otherwise

# this would not happen after removing this occurrence):

self.pool.check_spawn_psx_task(itask)

self.pool.remove(itask, 'request')

to_kill.append(itask)

itask.removed = True

itask.flow_nums.difference_update(fnums_to_remove)

matched_task_ids = {

*removed.keys(),

*((cycle, task) for task, cycle in inactive),

}

for cycle, task in matched_task_ids:

This was my intention to begin with, however it ended up being cleaner IMO to work with the ID as the ID is what is being used for the majority of things

The code I'm highlighting here converts from components to IDs back to components, i.e. the IDs are an unnecessary intermediary.

Otherwise, use Tokens objects which can do the conversion for you.

cylc/flow/scheduler.py

hjoliver · 2024-11-26T22:23:45Z

Note: We are only talking about SoD pre-spawning here (as we only remove children with partially satisfied prereqs).

SoD pre-spawning is really an implementation detail, so we could argue that these tasks should not be logged (I completely understand why Ronnie has logged this at DEBUG level). However, on the other hand, the n-window is now part of the model, it defines what the GUI sees and what commands users can issue against the task, so it is pertinent information.

Given that we log tasks that are pre-spawned by SoD at the INFO level, I agree that we should log their removal at the INFO level too

Yes, my thinking exackery.

Except that I sort of disagree that pre-spawning is just an implementation detail. Pre-spawned tasks are visible in n=0 because we decided to take partially satisfied prerequisites as an indicator of something wrong that should stall the workflow. And given that they are visible in n=0, their removal should be logged visibly too.

hjoliver

I'm approving this sucker, conditional on the small logging change. Nice work 🎖️

MetRonnie · 2024-11-27T15:58:58Z

If downstream tasks get removed, I think that should be logged at INFO level rather than DEBUG. It's potentially quite important information.

... (I completely understand why Ronnie has logged this at DEBUG level). However, on the other hand...

N.B. The logging at DEBUG level is not something I have touched; this is the default for TaskPool.remove(itask), unless the task is preparing, submitted or running in which case it becomes WARNING.

But I'm happy to change to make it INFO for the case in question

cylc/flow/flow_mgr.py

cylc/flow/scheduler.py

wxtim

I've had a decent look at the code, and made a real effort to break it in practice, without success. 🎉

oliver-sanders · 2024-11-28T13:32:38Z

Just had a shock reading back through the proposal:

At present (8.1.4) the Cylc 8 intervention is:

8.1.4!

Plus ensure traceback for internal errors when cleaning gets logged in verbose mode

oliver-sanders

Nice 🎉!

This is a biggie, well done @MetRonnie!

MetRonnie added 20 commits November 11, 2024 10:47

Cache (de)serialise functions

efcc5b5

`json.dumps()`/`json.loads()` are relatively slow (~1us). But these functions are likely to be called many times with `flow={1}`.

Tidy

9686d7e

Improve type safety for commands

52f6836

Tests: simplify fixture usage

61e834d

Stop reflog from hanging in some cases

49f4c5e

cylc remove removes tasks from flows in task_states & task_outputs …

5eee0ef

…tables

cylc remove unsets any prereq satisfaction provided by the removed …

9d562ae

…tasks in the given flow(s)

Fix nosec comments

265047d

Tidy

b873f2f

Remove no-longer valid test & clarify new behaviour

32e4810

cylc remove: update data store with removed flow nums

50c54b1

Integration tests: shorter workflow IDs

67ceba9

Ensure cylc remove unqueues tasks if it caused them to have unsatis…

61a803b

…fied prereqs

cylc remove: more efficiently unset prereq satisfaction of downstre…

59f5b14

…am tasks

Add tests

951b26f

Fix TaskProxy.match_flows() returning same reference as `self.flow_…

431d4ca

…nums`

Move cylc remove method from task pool to scheduler

55a4150

This will allow it to call the method to kill submitted/running tasks

Tidy/add type annotations

5b532b7

cylc remove kills running/submitted tasks

b7eaac0

MetRonnie added this to the 8.4.0 milestone Nov 11, 2024

MetRonnie requested review from hjoliver, oliver-sanders and wxtim November 11, 2024 11:58

MetRonnie self-assigned this Nov 11, 2024

MetRonnie force-pushed the cylc-remove branch from 4509bbf to 7a651e7 Compare November 11, 2024 11:58

MetRonnie mentioned this pull request Nov 11, 2024

Make cylc remove flow-aware and extend to historical tasks #6370

Closed

11 tasks

hjoliver reviewed Nov 12, 2024

View reviewed changes

This was referenced Nov 12, 2024

cylc remove: Address review and clear up confusing terminology in prerequisite.py MetRonnie/cylc-flow#34

Closed

Kill tasks during job prep. #5749

Closed

oliver-sanders reviewed Nov 26, 2024

View reviewed changes

hjoliver approved these changes Nov 27, 2024

View reviewed changes

wxtim reviewed Nov 28, 2024

View reviewed changes

cylc/flow/flow_mgr.py Show resolved Hide resolved

wxtim reviewed Nov 28, 2024

View reviewed changes

cylc/flow/scheduler.py Outdated Show resolved Hide resolved

wxtim reviewed Nov 28, 2024

View reviewed changes

cylc/flow/scheduler.py Show resolved Hide resolved

wxtim reviewed Nov 28, 2024

View reviewed changes

cylc/flow/scheduler.py Show resolved Hide resolved

wxtim approved these changes Nov 28, 2024

View reviewed changes

MetRonnie added 9 commits November 28, 2024 14:43

Ensure removed tasks that get killed are updated in the DB

95591ba

Fix cylc clean failing on simulation mode platform lookup

98181b6

Plus ensure traceback for internal errors when cleaning gets logged in verbose mode

Rewrite docs

91b40bc

Changelog

2c6b34a

De-flakify test

4742817

Prerequisite class: clear up confusing terminology

9b550c9

Tidy

0df68da

Improve tests

facf957

Add test

3eb6fb0

MetRonnie force-pushed the cylc-remove branch from 7a9d253 to 3eb6fb0 Compare November 28, 2024 14:44

MetRonnie added 2 commits November 28, 2024 15:18

Log removal from pool due to prereq being removed at INFO level

a567000

Merge remote-tracking branch 'upstream/master' into cylc-remove

7d91a99

oliver-sanders mentioned this pull request Nov 28, 2024

remove: handle reload #6502

Closed

oliver-sanders approved these changes Nov 28, 2024

View reviewed changes

oliver-sanders merged commit 9ff50f8 into cylc:master Nov 28, 2024
27 checks passed

MetRonnie deleted the cylc-remove branch November 28, 2024 18:46

oliver-sanders mentioned this pull request Nov 29, 2024

changes: remove cylc/cylc-doc#782

Closed

MetRonnie mentioned this pull request Nov 29, 2024

Skip Mode #6039

Merged

8 tasks

MetRonnie added the schema change Change to the Cylc GraphQL schema label Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `cylc remove` proposal #6472

Implement `cylc remove` proposal #6472

MetRonnie commented Nov 11, 2024 •

edited

Loading

hjoliver left a comment

oliver-sanders commented Nov 26, 2024 •

edited

Loading

oliver-sanders Nov 25, 2024

MetRonnie Nov 27, 2024 •

edited

Loading

oliver-sanders Nov 28, 2024 •

edited

Loading

hjoliver commented Nov 26, 2024 •

edited

Loading

hjoliver left a comment

MetRonnie commented Nov 27, 2024 •

edited

Loading

wxtim left a comment

oliver-sanders commented Nov 28, 2024

oliver-sanders left a comment

Implement cylc remove proposal #6472

Implement cylc remove proposal #6472

Conversation

MetRonnie commented Nov 11, 2024 • edited Loading

Summary

Flow numbers

Historical tasks

Kill submitted/running tasks

Check List

Footnotes

hjoliver left a comment

Choose a reason for hiding this comment

oliver-sanders commented Nov 26, 2024 • edited Loading

oliver-sanders Nov 25, 2024

Choose a reason for hiding this comment

MetRonnie Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

hjoliver commented Nov 26, 2024 • edited Loading

hjoliver left a comment

Choose a reason for hiding this comment

MetRonnie commented Nov 27, 2024 • edited Loading

wxtim left a comment

Choose a reason for hiding this comment

oliver-sanders commented Nov 28, 2024

oliver-sanders left a comment

Choose a reason for hiding this comment

Implement `cylc remove` proposal #6472

Implement `cylc remove` proposal #6472

MetRonnie commented Nov 11, 2024 •

edited

Loading

oliver-sanders commented Nov 26, 2024 •

edited

Loading

MetRonnie Nov 27, 2024 •

edited

Loading

oliver-sanders Nov 28, 2024 •

edited

Loading

hjoliver commented Nov 26, 2024 •

edited

Loading

MetRonnie commented Nov 27, 2024 •

edited

Loading