Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wait for preparing tasks to submit before auto restart #5062

Merged
merged 2 commits into from
Aug 16, 2022

Conversation

MetRonnie
Copy link
Member

@MetRonnie MetRonnie commented Aug 12, 2022

Fix bug where any preparing local jobs would get orphaned during the shutdown before an auto restart.

You could reproduce this bug by changing

diff --git a/tests/functional/restart/42-auto-restart-ping-pong.t b/tests/functional/restart/42-auto-restart-ping-pong.t
index 5f897b33e..ecf58e99f 100644
--- a/tests/functional/restart/42-auto-restart-ping-pong.t
+++ b/tests/functional/restart/42-auto-restart-ping-pong.t
@@ -74,5 +74,5 @@ log_scan2() {
 }
 
-EARS=5  # number of times to bounce the workflow between hosts
+EARS=40  # number of times to bounce the workflow between hosts
 NO_TESTS="$(( EARS * 5 + 1 ))"
 set_test_number "${NO_TESTS}"

and running the test, eventually it would orphan a preparing job and cause the test to fail.

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • No dependency changes
  • Tests are included
  • CHANGES.md entry included if this is a change that can affect users
  • No docs needed
  • If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

@MetRonnie MetRonnie added the bug Something is wrong :( label Aug 12, 2022
@MetRonnie MetRonnie added this to the cylc-8.0.1 milestone Aug 12, 2022
@MetRonnie MetRonnie self-assigned this Aug 12, 2022
Copy link
Member

@oliver-sanders oliver-sanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

  • Tested by breaking the job-submission pipeline after the "prep" stage to ensure auto-restart is delayed by preparing jobs.
  • Ran the functional/flakyfunctional restart tests locally (not run by CI).

@oliver-sanders
Copy link
Member

(could do with a changelog entry for this one)

@wxtim wxtim self-requested a review August 16, 2022 08:37
Comment on lines +1294 to +1296
log = LOG.debug
if self.options.reftest or self.options.genref:
log = LOG.info
Copy link
Member

@wxtim wxtim Aug 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
log = LOG.debug
if self.options.reftest or self.options.genref:
log = LOG.info
log = LOG.info if (self.options.reftest or self.options.genref) else LOG.debug

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't fit on 1 line which is why I left it the way it is

@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.1, 8.0.2 Aug 16, 2022
@wxtim wxtim merged commit 479ef0a into cylc:8.0.x Aug 16, 2022
@MetRonnie MetRonnie modified the milestones: 8.0.2, cylc-8.0.1 Aug 16, 2022
@MetRonnie MetRonnie deleted the auto-restart-orphan branch August 16, 2022 10:22
wxtim added a commit to wxtim/cylc that referenced this pull request Aug 18, 2022
* master:
  tui: add poll mutation (cylc#5075)
  bump dev version
  Add workflow field to ClientError, ClientTimeout
  Bump dev version
  Lint.hardcode style index numbers (cylc#5055)
  Wait for preparing tasks to submit before auto restart (cylc#5062)
  update changelog
  Prepare release 8.0.1
  scan: ignore FileNotFoundError (cylc#5065)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :(
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants