Is it possible to require all GitHub Actions tasks to pass without enumerating them?

[jameslamb]

The Question

Is it possible to configure required status checks in a repo to say “all GitHub Actions tasks must pass”, instead of needing to select individual GitHub Actions tasks by name?

This is how Travis, CircleCI, AppVeyor, and Azure DevOps integrations with GitHub all work…the default behavior is that all tasks must pass, and you can selectively mark tasks as optional.

I’d like to be able to have an administrator set this one time and not have to change it every time we change our GitHub Actions flow.

Background Context

Consider the screenshot below:

image924×284 40.3 KB

This is taken from a pull request in a project I’m working on. The master branch of that repository has a GitHub Actions workflow with 4 tasks. A repo administrator has manually marked each of them as required, following the steps in “About required status checks”. You can see, for example, that a task r-package (macOS-latest, clang) is required.

The pull request I’m proposing changes that GitHub Actions workflow, such that there is no longer a task called r-package (macOS-latest, clang). I am now in a situation where the pull request can’t be merged by maintainers because it looks like the required tasks never ran, and there a bunch of new non-required tasks that passed. See microsoft/LightGBM#3065 (comment) for more background.

Thanks for the time and consideration!

[msfjarvis]

Unfortunately there isn’t a way to do this. I’ve had this issue in the past as well where I had a path exclusion filter set up to not run build workflows on non-code changes and they similarly blocked merges. I had to end up doing this as a workaround.

[webknjaz]

@msfjarvis I've generalized this as https://github.com/marketplace/actions/alls-green#why a while back — maybe you could update your answer mentioning it?

Here's another answer from back then: https://github.com/orgs/community/discussions/26733#discussioncomment-3253155.

[jameslamb]

oh sad. Yeah I guess it is just something we’ll have to deal with. Not a big deal! Overall I’m still really liking GitHub Actions.

Thanks for the help.

[graingert]

@jameslamb I think another work around would be to create a job that “needs” the previous job:

jobs:
    test:
        matrix: ...
    allgood:
        needs: [test]

https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds

but it costs an extra worker

[jameslamb]

oooo cool idea! That’s very clever, I like it. I think that would solve what I’m looking for, thank you.

[graingert]

here’s the PR [ci] add an allgood job for easy [required] builds by graingert · Pull Request #3351 · microsoft/LightGBM · GitHub

[webknjaz]

@graingert it was discovered that when dependencies don’t succeed, that check job is marked as skipped which auto-merge treats the same way as success. I solved this problem with https://github.com/re-actors/alls-green.

Here's a short example:

---

on:
  pull_request:
  push:

jobs:
  tests:
    ...
    strategy:
      matrix:
        python-version:
        # - ~3.12.0-0  # doesn't yet exist as of Oct 04, 2022
        - ~3.11.0-0
        - >-
          3.10
        - 3.9
    ...
    continue-on-error: >-  # this allows "unstable" versions to fail, except for 3.11, meaning 3.12 alpha when it's added
      ${{
        (
          startsWith(matrix.python-version, '~')
          && !startsWith(matrix.python-version, '~3.11')
        )
        && true
        || false
      }}
    ...
    steps:
    - ...

  check:  # This job does nothing and is only used for the branch protection
    if: always()

    needs:
    - tests

    runs-on: ubuntu-latest

    steps:
    - name: Decide whether the needed jobs succeeded or failed
      uses: re-actors/alls-green@release/v1
      with:
        jobs: ${{ toJSON(needs) }}

...

See https://github.com/marketplace/actions/alls-green#why.

[jameslamb]

Ah! After adding the following to the workflow YAML:

if: always()

I saw the behavior you described...a failing required job caused the catch-all job to marked as failed, which blocked merging.

I know that if: always() was in the doc you linked, sorry for missing it.

I agree now, this action definitely solves the original issue I asked about more completely than what can be achieved with GitHub Actions' current behavior and the earlier suggestions in this discussion. I've modified this to mark your suggestion as the answer.

Thanks for your work on https://github.com/re-actors/alls-green and for the help!

[webknjaz]

You're welcome :)

[nikaro]

This is a very cool Action, thanks for that. But it works only for the jobs of one workflow. In the case where multiple workflows are triggered, and you want all of them to pass, currently there is no solution.

[webknjaz]

@nikaro The solution is to use reusable workflows or have the action integrated into several workflows under different names and all added to the branch protection. I should probably document that.
I've also seen actions that allow polling to wait other checks so that's another way.
My preference is using the reusable workflows:

1. It allows to modularize different job definition, separating them logically even within the same repo
2. I came up with an approach of having such workflows prefixed with reusable- for visual distinction in the file list. They also only have workflow_call: trigger, no others.
3. I tend to call the main workflow definition ci-cd.yml and that's where I add alls-green. If it only consists of the other workflow calls, it's easier to reason about structuring the job dependencies on the high level. It also lets me avoid having a million lines of YAML in a single file.
4. Bonus: it's possible to make a separate workflow that runs on cron and include that ci-cd.yml (this is a good workaround for low-activity repos where the workflows with cron tend to become disabled periodically, so it wouldn't affect the PRs)

[webknjaz]

@nikaro some of the above ideas are documented here: https://learn.scientific-python.org/development/guides/gha-basic/#reusable-workflows.

[anmolchopra69]

How can we add/remove Details link ?
How can we re-direct to some url using Details link ?

[webknjaz]

That would require a full GitHub App implementation, that's beyond what GitHub Actions CI/CD can provide you with. Though, https://github.com/marketplace/actions/alls-green does print out some rich output via a Job Summary interface (it shows up on the workflow summary page, below the job relationships graph).

[adamu]

I run an app, Mergery, that does this.

[webknjaz]

Yes, some apps handle this properly. I think, Mergify does that too. However, this question seems to be more about GHA rather than external apps.

[adamu]

Not at all. In fact if you use Mergery it requires no code to be committed to set up, and because it's an app it avoids the catch-22 of wanting all actions to pass, but doing that check itself in an action.

[webknjaz]

Your statements seem to contradict each other, though.

[webknjaz]

I don't criticize, I pointed out above that such an app exists. And it's closed source too. Perhaps your wording confused me there.
It'd be nice to have this feature in GH, of course. Since it's not, my style is to make configs as code — straightforward and transparent. I know that apps can go beyond that since they can poll statuses from the old Statuses API as well as Checks API, which I didn't implement in the action, but maybe at some point I will.

[beatngu13]

This is really a problem, especially for large organizations/enterprises. AFAIK it is currently only possible to manually select status checks via "Require status checks to pass" at the repository or organization level, as part of a ruleset. An additional checkbox "Require all status checks to pass" would help a lot.

[beatngu13]

And regarding the accepted answer re-actors/alls-green: it doesn't seem maintained (last commit 2 years ago)? But even if, I believe GitHub should a provide a native solution.

[danielsiwiec]

yep, this is shocking you can't require all configured checks to pass. I'd also expect this feature to be available