Configurable grace period for cancel-in-progress jobs

[briceburg]

Select Topic Area

Question

Body

Jobs in concurrency groups cancelled by the cancel-in-progress: true mechanism are terminated without the ability to cleanup after themselves. This discussion provides a good summary.

The eager termination makes it unsafe to execute terraform, for instance, in a cancelable job -- which would normally be able to cleanly shut down given a SIGINT and adequate grace period.

As a fix, can we have a configurable grace period for cancel-in-process jobs? Something like;

jobs:
  deploy:
    concurrency:
      group: deploy-${{ inputs.environment }}
      cancel-in-progress: true
      cancel-in-progress-grace-period: 60000

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency

parameter	default	description
cancel-in-progress-grace-period	7500	Time in milliseconds to allow cancelled jobs to respond to SIGINT. Max 60000 (1m).

[ringerc]

A configurable grace period would be very helpful, but definitely does not fully solve the issues with writing reliable, robust github actions runner cancellation logic.

If the concurrency group runs a shell: bash step, child processes run by the shell will never see any SIGINT or SIGTERM sent to the shell anyway. So they'll get no chance to clean up. This can be worked around by exec'ing the child process, e.g.

steps:
  - name: foo
    shell: bash
    run: |
       exec terraform ...

but very few people will do that, and by doing so you lose the ability to have logic after the "critical" command in your step.

This is because child processes of a step's top-level process are not sent any SIGINT or SIGTERM by the github actions runner when the step is cancelled. It only signals the top level process. So unless the top-level process propagates a signal, the step's other processes will get no warning of a step cancellation and no chance to clean up. Because bash doesn't propagate signals to child processes while it's blocked waiting for them to exit and bash is the default shell, the vast majority of github processes started by actions steps will never get any signals on cancellation.

Also, if processes started by a step detach from the session-leader, they continue to run even after the step is cancelled, so they can still be running when any if: always() cleanup step runs. It's easy to accidentally detach processes from the session leader ("daemonize") them in bash especially when trying to do things like run a command that writes to a log file on disk while tailing its logs to stdout. This means a cleanup step cannot assume all processes from the prior steps have really terminated. I haven't tested if this applies to concurrency groups but it probably does.

See the demos I wrote on https://github.com/ringerc/github-actions-signal-handling-demo

In the end anything running on github actions must have a way to recover if the actions worker node dies abruptly and without warning while it's in the middle of running a workload. But that should be a lot rarer than things like cancellations.

It'd help if:

• the docs explained the cancellation logic, time limits, signals delivered, etc
• an option was available at job and step level to tell the github actions runner to signal the whole process group and its session under the step being cancelled, not just the leader. It should not be made the default as it could break existing workflows that correctly propagate signals to child processes, possibly with translation logic etc.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬