Adding new choice to --on-error

[AlexTate]

Summary

This pull request introduces a new choice, kill, for the --on-error parameter.

Motivation

There currently isn't a way to have cwltool immediately stop parallel jobs when one of them fails. One might expect --on-error stop to accomplish this, but the help string is specific and accurate: "do not submit any more steps". Since scatter and subworkflow are treated as single "steps" within the parent workflow, this means cwltool is not wrong to wait for the rest of the step's parallel jobs to finish when --on-error stop. However, sometimes individual scatter jobs take a long time to complete, so if one of them fails early on, cwltool might wait great lengths of time for the other scatter jobs to complete before terminating the workflow. With --on-error kill, all running jobs are quickly notified and self-terminate upon one job's failure.

Demonstration of the Issue

When running the following workflow with cwltool --parallel --on-error stop, the total runtime is ~33 seconds despite one of the scatterstep tasks terminating unexpectedly. Ideally the workflow would terminate immediately. --on-error kill accomplishes that.

#!/usr/bin/env cwl-runner

class: Workflow
cwlVersion: v1.2

inputs:
  sleeptime:
    type: int[]
    default: [ 33, 33, 33, 33, 33 ]
outputs: { }
requirements:
  - class: ScatterFeatureRequirement

steps:
  scatterstep:
    in: { sleeptime: sleeptime }
    out: [ ]
    scatter: sleeptime
    run:
      class: CommandLineTool
      baseCommand: sleep
      inputs:
        sleeptime: { type: int, inputBinding: { position: 1 } }
      outputs: { }
  kill:
    in: { }
    out: [ ]
    run:
      class: CommandLineTool
      baseCommand: [ 'bash', '-c' ]
      arguments:
        - |
          # Wait 1 second for scatter to spin up then select a random sleep process to kill
          sleep 1
          ps -ef | grep 'sleep 33' | grep -v grep | awk '{print $2}' | shuf | head -n 1 | xargs kill -9
      inputs: { }
      outputs: { }

Forum Post

https://cwl.discourse.group/t/how-to-fail-fast-during-parallel-scatter/868

Concerns

• workflow_eval_lock.release() had to be moved to the finally block in MultithreadedJobExecutor.run_jobs()
• Are any important steps skipped in JobBase._execute() due to if runtimeContext.kill_switch.is_set(): return? For that matter, shouldn't there be a finally block to contain some of these steps such as deleting runtime-generated files containing secrets? Update Nov 13: these post-subprocess tasks were moved into a finally block in JobBase._execute() to ensure that they aren't skipped by jobs setting or responding to the kill switch. See abc4c3f.
• The kill switch response in TaskQueue is fairly loose. Since the response is primarily handled at the job level, any tasks that start after the kill switch is activated will take care of themselves and self terminate Update Nov 13: TaskQueue response to the kill switch was tightened in b302eca. Still, a race condition exists where a job may be started within a narrow window of time after the kill switch has been set, but if that happens the leaked job will still self terminate within the monitor function's polling interval (currently 1 second).

[cwl-bot]

This pull request has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/how-to-fail-fast-during-parallel-scatter/868/5

[codecov]

Codecov Report

Attention: Patch coverage is 67.58621% with 47 lines in your changes missing coverage. Please review.

Project coverage is 80.53%. Comparing base (e4d42b8) to head (e139fae).

Files with missing lines	Patch %	Lines
cwltool/job.py	69.51%	18 Missing and 7 partials ⚠️
cwltool/workflow_job.py	63.63%	4 Missing and 4 partials ⚠️
cwltool/workflow.py	61.53%	5 Missing ⚠️
cwltool/executors.py	63.63%	2 Missing and 2 partials ⚠️
cwltool/task_queue.py	55.55%	3 Missing and 1 partial ⚠️
cwltool/errors.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1974      +/-   ##
==========================================
- Coverage   83.93%   80.53%   -3.41%     
==========================================
  Files          46       46              
  Lines        8312     8396      +84     
  Branches     1959     1973      +14     
==========================================
- Hits         6977     6762     -215     
- Misses        856     1072     +216     
- Partials      479      562      +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[mr-c]

Thank you, again, @AlexTate for your PR!

tests/test_parallel.py::test_on_error_kill is unfortunately failing.

[mr-c]

Thank you again for this contribution, @AlexTate ! Alas, the test is sometimes hanging in CI: https://github.com/common-workflow-language/cwltool/actions/runs/11496251742/job/31997507449?pr=1974#step:8:1983

[mr-c]

Almost there!

Check make diff_pydocstyle_report or tox -e py312-pydocstyle