re-implement the task retry state using xtriggers

[oliver-sanders]

• Remove the following task states:
  • TASK_STATUS_RETRYING
  • TASK_STATUS_SUBMIT_RETRYING
• Re-implement retry logic using xtriggers.
  • Create an xtirigger called cylc[_submission]_retry_<task>.<cycle>
    • Use the cylc namespace for special triggers.
  • Create only one xtrigger per task per retry type.
• Reset "retrying" tasks to waiting.

Caveats:

• Can no longer select tasks by retrying states on the CLI (e.g. <cyclepoint>/*:retrying)
• Retrying tasks are now waiting so have a different position in TASK_STATUS_ORDERED
  • TODO need to check this doesn't break polling logic

These changes partially address cylc/cylc-admin#47

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included.
• I have opened a documentation Issue at Document how retries work in Cylc 8 cylc-doc#162.

[hjoliver]

That was quick!

[hjoliver]

I'm part way through reviewing this. I think it's all good, but there's a few caveats to think through, as you've pointed out.

[hjoliver]

Looks good, tests as working. Trying to think through the polling logic question...

[kinow]

bash failures are related to docker and memory, see #3856 for fix

2020-10-07T04:25:41.9392152Z    write (2, "virtual memory exhausted\n", 25);

[hjoliver]

All functional and unit tests passing now.

The bash test tests/functional/broadcast/00-simple.t fails intermittently on other branches too.

[hjoliver]

(The last 3 commits just modify comments, via the GH UI "commit suggestion" button).

[hjoliver]

(All tests passing 😁 )

[kinow]

A few typos and pydocs issues. No major issues with rest of the code (unless that change in rundb is a legit issue). I did find a few interesting things in other parts of the code, but they were not in this pull request (which means I've got more post-its on my screen now to investigate later 😬 )

@hjoliver I tested it with Cylc UIServer, and found no exceptions in the logs. With Cylc UI, however, the complex workflow logged that error due to missing parents.

I tested master and the errors didn't occur. But given how little we know about that error, it could be that the same happens on master, but intermittently. I have no idea if complex has anything special about it with regards to the change of retry state in this PR 👍

Thanks!
B

[hjoliver]

I have no idea if complex has anything special about it with regards to the change of retry state in this PR

I just checked, it doesn't look like any tasks in the complex suite will retry so it must be a coincidence.

[kinow]

Had a look at the files that GH UI showed as changed since my last review (thanks GH for such handy functionality). Looking good to me, +1

[hjoliver]

@kinow - the same flaky bash test failed again. I think you can approve and merge this if the functional tests all pass.

[kinow]

(@hjoliver I'm guessing CI is still a bit unstable, but approved since the conflict changed files were only 3, from a PR that I reviewed yesterday; no issues that I could see)

[kinow]

@kinow - the same flaky bask test failed again. I think you can approve and merge this if the functional tests all pass.

Missed this comment by 23 seconds, while I was typing. But I managed to approve before your comment 😄

[hjoliver]

(Two approvals 😁 and the bash test fail is a known flaky ... merging)

[oliver-sanders]

Wow, did that really happen.