success() returns false if dependent jobs are skipped

[suzuki-shunsuke]

Select Topic Area

Bug

Body

Please see the following workflow definition and result.

• test success() suzuki-shunsuke/test-github-action#119
• https://github.com/suzuki-shunsuke/test-github-action/actions/runs/3979279366
• https://github.com/suzuki-shunsuke/test-github-action/actions/runs/3979279366/workflow

---
name: Test
on: pull_request
jobs:
  foo:
    runs-on: ubuntu-latest
    steps:
      - run: echo hello
  bar:
    runs-on: ubuntu-latest
    if: "false"
    steps:
      - run: echo bar
  zoo:
    runs-on: ubuntu-latest
    if: "success()"
    needs:
      - foo
      - bar
    steps:
      - run: echo hello

The job zoo depended on foo and bar and foo succeeded and bar was skipped, then zoo was skipped.

But I think zoo should have been run because none of the dependent jobs failed or was canceled.

https://docs.github.com/en/actions/learn-github-actions/expressions#success

Returns true when none of the previous steps have failed or been canceled.

https://docs.github.com/en/actions/learn-github-actions/contexts#needs-context

needs.<job_id>.result | string | The result of a job that the current job depends on. Possible values are success, failure, cancelled, or skipped.

skipped is different from cancelled.

I changed if from success() to ! failure(), then it worked as expected.

[suzuki-shunsuke]

I created a repository to reproduce this issue.

https://github.com/suzuki-shunsuke/bug-reproduction-github-actions-success

[suzuki-shunsuke]

I created a GitHub Support ticket and got a reply today.

Thank you for writing to GitHub Support.

It appears the documented behavior is only being implemented if the success() function is used at the step level.
I have opened an issue with Engineering to seek further clarification. I will let you know as soon as I have an update from them.

I'm waiting for further reply from support.

[ChristopherHX]

Let's see if the GitHub Engineering alters the behavior of success() to include skipped jobs after more than 3 years of GitHub Actions formerly based on Azure Pipelines (which has this (unexpected) behavior documented https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#succeeded).

There is an very old issue about this behavior (from 2020): actions/runner#491 (comment)

You should use !failure() && !cancelled() as a workaround, unless you want that pressing cancel does nothing.

[suzuki-shunsuke]

Oh, thank you for sharing. I didn't know the issue.

[suzuki-shunsuke]

#45058 (comment)

I'm waiting for further reply from support.

I got a reply from support.

Hi Shunsuke,

Apologies for taking a while to get back to you on this.

The GitHub Actions team have placed this issue on the backlog. This is something that they are looking to fix but I do not have a timeline for when that will be done, sorry. I will open a follow up ticket to let you know as soon as the fix is deployed.

For now, as you noted, using the ! failure() workaround would achieve the equivalent behavior.

Let me know if you have any further questions.

[lyuhau]

I'm having similar issues. A job with if-condition always() && !contains(needs.*.result, 'skipped') gets skipped even though that expression evaluates to true. always() && !contains(needs.*.result, 'failure') on the other hand works as expected. Here's the sample workflow file that I made to test:

name: ci

on:
  pull_request: null
  workflow_dispatch: null

jobs:
  runs:
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  skips:
    if: false
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  runs-with-skipped-upstream:
    needs: skips
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  always-and-no-skips:
    needs: [runs, runs-with-skipped-upstream]
    if: always() && !contains(needs.*.result, 'skipped')
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  always-and-no-failures:
    needs: [runs, runs-with-skipped-upstream]
    if: always() && !contains(needs.*.result, 'failure')
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  always-and-no-skips-or-failures:
    needs: [runs, runs-with-skipped-upstream]
    if: always() && !contains(needs.*.result, 'skipped') && !contains(needs.*.result, 'failure')
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.job }}

  debug-print:
    needs: [runs, runs-with-skipped-upstream]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: |
          echo "toJSON(needs) = ${{ toJSON(needs) }}"
          echo "toJSON(needs.*.result) = ${{ toJSON(needs.*.result) }}"
          echo "!contains(needs.*.result, 'skipped') = ${{ !contains(needs.*.result, 'skipped') }}"
          echo "!contains(needs.*.result, 'failure') = ${{ !contains(needs.*.result, 'failure') }}"

And the repository: https://github.com/lyuhau/github-actions-needs-if-bug

[lyuhau]

The console of the debug-print job:

[amanfredi]

Please change this bonkers behavior

[cundejo]

Please fix this, it doesn't make sense at all. Thank you!

[LudovicTOURMAN]

Since I struggled with some weird behavior (and not only with the success() function), I wanted to share a result table of the check functions.

flowchart LR;
    A[Job A]-->B1[Job B1];
    A-->B2[Job B2];
    B1-->D[" "];
    B2-->D;
    D-->E["always()"];
    D-->F["success()"];
    D-->G["failure()"];
    D-->H["cancelled()"];

Loading

Job A	Job B1 (needs A)	Job B2 (needs A)	always() (needs A + B1 + B2)	success() (needs A + B1 + B2)	failure() (needs A + B1 + B2)	cancelled() (needs A + B1 + B2)
${\color{green}\text{success}}$	${\color{green}\text{success}}$	${\color{green}\text{success}}$	${\color{green}\text{true}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$
${\color{green}\text{success}}$	${\color{grey}\text{skipped}}$	${\color{green}\text{success}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$
${\color{green}\text{success}}$	${\color{red}\text{failure}}$	${\color{green}\text{success}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$
${\color{green}\text{success}}$	${\color{grey}\text{skipped}}$	${\color{orange}\text{cancelled}}$ (concurrency)	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$
${\color{green}\text{success}}$	${\color{red}\text{failure}}$	${\color{orange}\text{cancelled}}$ (concurrency)	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$
${\color{green}\text{success}}$	${\color{grey}\text{skipped}}$	${\color{orange}\text{cancelled}}$ (manual)	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$
${\color{green}\text{success}}$	${\color{red}\text{failure}}$	${\color{orange}\text{cancelled}}$ (manual)	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$
${\color{red}\text{failure}}$	${\color{grey}\text{skipped}}$	${\color{grey}\text{skipped}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$
${\color{grey}\text{skipped}}$	${\color{green}\text{success}}$	${\color{green}\text{success}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$
${\color{grey}\text{skipped}}$	${\color{red}\text{failure}}$	${\color{green}\text{success}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$
${\color{orange}\text{cancelled}}$ (concurrency)	${\color{grey}\text{skipped}}$	${\color{grey}\text{skipped}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$
${\color{orange}\text{cancelled}}$ (manual)	${\color{orange}\text{cancelled}}$	${\color{orange}\text{cancelled}}$	${\color{green}\text{true}}$	${\color{red}\text{false}}$	${\color{red}\text{false}}$	${\color{green}\text{true}}$

⚠️ Notice how different are the results between a workflow cancelled manually and a workflow cancelled by concurrency mechanism (cancel-in-progress: true)

I hope that could help people dig into weird behaviors and make GitHub fixing/improving those weird condition behaviors (non-sense from a developer's pov)

[EcksDy]

Amazing, thank you for sharing!

[Felixoid]

Here's my way to check if all jobs in the needs have succeeded

if: ${{ ! (contains(needs.*.result, 'skipped') || contains(needs.*.result, 'failure')) }}

I must say, it's very ugly. Instead of needs_succeeded(), we need to have such a boilerplate. Connecting to the fact that we don't have anchors actions/runner#1182, it's just a shame.

for the reference Felixoid/actions-experiments#9