validate: add separate option for cyclic graph detection

[dpmatthews]

Cyclic graph validation can be quite slow and is currently included as part of --strict.
See cylc/cylc-admin#111 (comment)
See also #3866

[oliver-sanders]

Note: we could potentially skip cyclic graph detection based on an arbitrary limit to the number of tasks or dependencies.

[MetRonnie]

Something I'm a bit puzzled by: why does the _check_circular() method use [visualization]initial cycle point? Why [visualization]?

cylc-flow/cylc/flow/config.py

Lines 843 to 852 in 9b9dd86

	def _check_circular(self):
	"""Check for circular dependence in graph."""
	start_point_string = (
	self.cfg['visualization']['initial cycle point'])
	lhs2rhss = {} # left hand side to right hand sides
	rhs2lhss = {} # right hand side to left hand sides
	for lhs, rhs in self.get_graph_raw(
	start_point_string,
	stop_point_string=None,
	):

This also ties into making that section obsolete as part of #3696

[oliver-sanders]

The visualisation ICP/FCP (Cylc7) define the window that the cylc graph visualisation tool uses (this is a static visualisation tool used to assist with workflow design). So rather than displaying the entire workflow (note the ICP/FCM might not be hardcoded in the workflow anyway) the visualisation settings can define a bit of the workflow that you want to look at.

I guess that graph validation just piggy-backed on this when provided so it ran over a sensible sized chunk of the graph?

Here's the PR that introduced the current system, though the visualisation settings were in use long before: #2332.

[TomekTrzeciak]

How slow is too slow? We do some cycle detection with NetworkX on our suite and even on fairly large graphs this doesn't seem to be a problem. Eg. on a graph with 832 nodes it takes ~1 sec. and for 223987 nodes ~31 secs.

[oliver-sanders]

How slow is too slow?

<< 1 sec

The intention is to make validate light-weight, so the idea behind the 100-task limit was to run cyclic-graph detection only in trivial cases where we know it will run very quickly.

[TomekTrzeciak]

What happens if I run a suite with more than 100 tasks and circular dependencies? Does it just stall?

[MetRonnie]

Currently on master, with the following workflow:

[scheduling]
    cycling mode = integer
    [[graph]]
        R1 = a => b => c => d => a

it doesn't run anything:

INFO - Run: (re)start=0 log=1
INFO - Cylc version: 8.0a3.dev
INFO - Run mode: live
INFO - Initial point: 1
INFO - Final point: None
INFO - Suite shutting down - AUTOMATIC
INFO - DONE

[oliver-sanders]

(same situation as at present, cyclic graph detection is opt-in, currently via --strict)

[MetRonnie]

There isn't a --strict option for cylc run though?

[MetRonnie]

After discussing today, this behaviour of cylc run will remain unchanged