cylc clean 3: targeted clean #4237

MetRonnie · 2021-06-02T10:55:18Z

These changes partially address #3887

A targeted version of [cylc clean]...

Implement targeted clean:

  # Remove the workflow's log directory
  $ cylc clean foo/bar --rm log

  # Remove the log and work directories
  $ cylc clean foo/bar --rm log:work
  # or
  $ cylc clean foo/bar --rm log --rm work

  # Remove all job log files from the 2020 cycle points
  cylc clean foo/bar --rm 'log/job/2020*'

  # Remove all .csv files
  $ cylc clean foo/bar --rm '**/*.csv'

  # Only remove the workflow on the local filesystem
  $ cylc clean foo/bar --local-only

  # Only remove the workflow on remote install targets
  $ cylc clean foo/bar --remote-only

These changes also close #4133

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.py and
conda-environment.yml.
Appropriate tests are included (unit and functional).
Appropriate change log entry included.
Documentation will be handled by Document cylc clean cylc-doc#256

Add test for --remote-only option too

Also changed notation of symlink stuff from dst/src to path/target, because I get really confused by dst/src!

By setting them to empty string

Add more unit tests

MetRonnie · 2021-06-02T11:38:25Z

Fast tests failure on MacOS seems unrelated

cylc/flow/workflow_files.py

oliver-sanders

Looks good, run some basic tests, still need to get my head around what it's doing with symlinks.

Some small comments:

cylc/flow/exceptions.py

oliver-sanders · 2021-06-08T08:04:52Z

cylc/flow/exceptions.py

+    def __init__(self, exc: Exception) -> None:
+        CylcError.__init__(
+            self,
+            f"{exc}. This is probably a temporary issue with the filesystem, "
+            "not a problem with Cylc."
+        )


(minor comment more for future sake)

A pattern that is a bit nicer for Python interfaces and testing is to store exc as an attribute on the FileRemovalError and handle the representation in the __str__ method.

Will get around to refactoring the other errors one day when we have less to do 😆.

cylc/flow/pathutil.py

cylc/flow/workflow_files.py

This filters out all redundant matches, i.e. matches that are subpaths of other matches (except for the std symlink dirs - we need to return these as we need to follow their targets). As was implemented in a previous commit, it still also filters out subpaths of symlinks as we don't want to follow symlinks (again except for the std symlink dirs).

Add unit tests

MetRonnie · 2021-06-21T11:56:20Z

tests/unit/test_pathutil.py

@@ -44,7 +44,7 @@
    remove_dir_or_file
 )

-from tests.unit.conftest import MonkeyMock


Anyone know why this import stopped working suddenly on GH Actions? The absolute import is still working locally for me

_________________ ERROR collecting tests/unit/test_pathutil.py _________________ tests/unit/test_pathutil.py:47: in <module> from tests.unit.conftest import MonkeyMock E ModuleNotFoundError: No module named 'tests'

It might be the difference between the editable install you will likely have for local development and the regular install on GH actions. I think the editable install will put the tests folder in PYTHONPATH.

The tests folder shouldn't be imported directly like that (hasn't been configured as a Python package, has no __init__.py so will be interpreted as a namespace package?).

oliver-sanders

Gone through the changes since last review, looks good to me.

cylc/flow/workflow_files.py

tests/unit/test_workflow_files.py

Do not retry a failed clean on an install target using the next platform in the list if the remote clean command failed due to a non-SSH reason (if ret code != 255).

datamel

Yes, I'm happy with the change, thanks @MetRonnie.
I have minor comments (one note to self for my pr once this is merged), nothing that should affect merge.

datamel · 2021-06-28T09:03:20Z

cylc/flow/task_remote_cmd.py

@@ -83,29 +83,29 @@ def create_client_keys(srvd, install_target):
    os.umask(old_umask)


-def remote_init(install_target, rund, *dirs_to_symlink):
+def remote_init(install_target: str, rund: str, *dirs_to_symlink: str) -> None:


I have memories of @oliver-sanders saying that *args is a list?

*args is a tuple, **kwargs is a dict, how mypy works with this is a whole other kettle of fish.

I found an example in the docs which suggests that *args: str means that all *args should be str (i.e. the tuple thing is implicit).

https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#when-you-re-puzzled-or-when-things-are-complicated

datamel · 2021-06-28T12:58:30Z

cylc/flow/workflow_files.py

+        SHARE_CYCLE_DIR, SHARE_DIR, LOG_DIR, WORK_DIR, ''
+    ])
+    """The paths of the symlink dirs that may be set in
+    global.cylc[symlink dirs], relative to the run dir


This is changing into [install] symlink dirs but I can change it on my pr: #4250

oliver-sanders

(went through and resolved outstanding comments)

oliver-sanders

Tested as best I can, everything looks good to me, edge cases well handled.

MetRonnie added 13 commits June 2, 2021 11:47

Type annotations

988648f

cylc clean: add --remote-only option

506bb38

Fix & refactor cylc clean unit tests

eacdcce

Add test for --remote-only option too

cylc clean: Implement targeted clean

a037073

Fix error when share/cycle sylink dir same as share

505b27b

Also changed notation of symlink stuff from dst/src to path/target, because I get really confused by dst/src!

Allow symlink dirs in global config to be unset

262126e

By setting them to empty string

Fix cylc#4133 - better error handling when rmtree fails

b02d74f

Tidy

7e0ee50

cylc clean: Fix deletion of symlink dir targets in targeted clean

fa112a2

cylc clean: overhaul targeted clean to make globs work

4581158

Refactor cylc clean test into fixture

2268f20

cylc clean: Fix tidying of empty run dir parents

d7a9f82

Add more unit tests

cylc clean: add help examples

634109b

MetRonnie added this to the cylc-8.0b2 milestone Jun 2, 2021

MetRonnie requested review from oliver-sanders and datamel June 2, 2021 10:55

MetRonnie self-assigned this Jun 2, 2021

oliver-sanders mentioned this pull request Jun 4, 2021

pause: workflow cannot be stalled if it is paused #4244

Merged

9 tasks

MetRonnie commented Jun 4, 2021

View reviewed changes

cylc/flow/workflow_files.py Show resolved Hide resolved

oliver-sanders reviewed Jun 8, 2021

View reviewed changes

MetRonnie marked this pull request as draft June 10, 2021 10:09

MetRonnie added 4 commits June 11, 2021 12:53

Remote clean: iterate over proc queue more elegantly

a2fcaea

Update changelog

6eb3500

Address code review

82ec437

MetRonnie force-pushed the cylc-clean branch from 4396576 to 59ba89e Compare June 11, 2021 17:08

Move glob_run_dir() to workflow_files

011e6e0

Add unit tests

MetRonnie force-pushed the cylc-clean branch from 59ba89e to 011e6e0 Compare June 11, 2021 17:20

MetRonnie marked this pull request as ready for review June 11, 2021 17:29

MetRonnie added 3 commits June 21, 2021 10:43

Merge branch 'master' into cylc-clean

3a44181

Fix small mistake

2dc0f0d

Fix test failures

4f23670

MetRonnie force-pushed the cylc-clean branch from 2dc0f0d to 011e6e0 Compare June 21, 2021 11:47

MetRonnie commented Jun 21, 2021

View reviewed changes

oliver-sanders reviewed Jun 22, 2021

View reviewed changes

cylc/flow/workflow_files.py Outdated Show resolved Hide resolved

cylc/flow/workflow_files.py Outdated Show resolved Hide resolved

cylc/flow/workflow_files.py Outdated Show resolved Hide resolved

tests/unit/test_workflow_files.py Show resolved Hide resolved

Remote clean: only re-run on different platform if SSH failed (255)

fadda2a

Do not retry a failed clean on an install target using the next platform in the list if the remote clean command failed due to a non-SSH reason (if ret code != 255).

MetRonnie mentioned this pull request Jun 24, 2021

cylc clean #3887

Open

datamel approved these changes Jun 29, 2021

View reviewed changes

oliver-sanders reviewed Jun 29, 2021

View reviewed changes

oliver-sanders approved these changes Jun 29, 2021

View reviewed changes

oliver-sanders merged commit 6e28784 into cylc:master Jun 29, 2021

MetRonnie deleted the cylc-clean branch June 29, 2021 13:04

MetRonnie linked an issue Jun 29, 2021 that may be closed by this pull request

cylc clean #3887

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cylc clean 3: targeted clean #4237

cylc clean 3: targeted clean #4237

MetRonnie commented Jun 2, 2021 •

edited

Loading

MetRonnie commented Jun 2, 2021

oliver-sanders left a comment

oliver-sanders Jun 8, 2021

MetRonnie Jun 21, 2021 •

edited

Loading

oliver-sanders Jun 22, 2021

oliver-sanders left a comment

datamel left a comment

datamel Jun 28, 2021

oliver-sanders Jun 29, 2021

datamel Jun 28, 2021

oliver-sanders left a comment

oliver-sanders left a comment

cylc clean 3: targeted clean #4237

cylc clean 3: targeted clean #4237

Conversation

MetRonnie commented Jun 2, 2021 • edited Loading

MetRonnie commented Jun 2, 2021

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders Jun 8, 2021

Choose a reason for hiding this comment

MetRonnie Jun 21, 2021 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Jun 22, 2021

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

datamel left a comment

Choose a reason for hiding this comment

datamel Jun 28, 2021

Choose a reason for hiding this comment

oliver-sanders Jun 29, 2021

Choose a reason for hiding this comment

datamel Jun 28, 2021

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

MetRonnie commented Jun 2, 2021 •

edited

Loading

MetRonnie Jun 21, 2021 •

edited

Loading