Fix failure to clean on remote host with leftover contact file

[MetRonnie]

This is a small change with no associated Issue.

From Teams:

Cylc clean bug report... it looks like it is trying to perform the whole "is this workflow running" check on the remote side as well as the invocation side which isn't right:

$ cylc clean u-co906/run1
INFO - Cleaning u-co906/run1 on install target: ab
ERROR - Could not clean u-co906/run1 on install target: ab
   platform: abc - clean up did not complete
   COMMAND:
       ssh CYLC_VERSION=8.0rc4.dev CYLC_ENV_NAME=cylc-8.0rc4.dev-3 \
           bash --login -c 'exec "$0" "$@"' timeout 120 cylc clean \
           --local-only u-co906/run1
   RETURN CODE:
       1
   STDOUT:
   STDERR:
       CylcError: Cannot determine whether workflow is running on <scheduler-host>.
       ~fcm/cylc-8.0rc4.dev-3/bin/python3.9 ~fcm/cylc-8.0rc4.dev-3/bin/cylc play u-co906 --host=localhost

CylcError: Remote clean failed for u-co906/run1

Note: We cannot necessarily SSH back from remote platforms to the scheduler host.

Solution

If there us a leftover contact file when doing cylc clean, if the host field gives a host that is not the local host, it means we are doing re-invoked cylc clean --local-only on the remote host and we should not check if the scheduler process is still running (it can't be).

The workflow DB only exists on the scheduler host. So if we are doing cylc clean --local-only and there is a leftover contact file but no DB, it probably means this is cylc clean being re-invoked on a remote host and we can ignore the contact.

Even if the fix gets hit when someone genuinely someone runs cylc clean --local-only on the scheduler host, it shouldn't matter.

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg and conda-environment.yml.
• Appropriate tests are included (functional).
• Appropriate change log entry included.
• No documentation update required.

[MetRonnie]

Unrelated change but spotted this leftover today

[wxtim]

Happy with this - add a changelog entry before merging.

[MetRonnie]

Dang, just realised this approach is flawed - if there are configured remote run hosts, it will allow cylc clean to delete a running workflow

[MetRonnie]

Changed approach, relying on the fact that the workflow DB only exists on the scheduler host.

[MetRonnie]

Re-done as per Oliver's suggestion

[hjoliver]

(A very quick review but LGTM; I haven't tested it).

[wxtim]

I'm OK with this. @oliver-sanders - do you want to do a final check?

[oliver-sanders]

LGTM