Properly show continue-on-error jobs/steps in PR UI

[eregon]

continue-on-error is a very useful part of GitHub Actions, but unfortunately it's reflected extremely poorly (= not at all) in the UI and specifically in the pull request UI.
That means it's near-useless or barely better than || true, which is not the desired feature.
The desired feature is to allow some jobs to fail, and indicate it on the PR UI in a sensible way, much like TravisCI did with allow-failures: true.

continue-on-error:

• if used at the step level, then everything appears green and the only way to notice the failure would be to manually open the step and carefully read the output in details. There is also an annotation Check failure on line 1 in .github but that doesn't link to the proper place.
  That's not enough notification, and there seem to be no reasonable way to see if that job actually passed or failed.
  Example: https://github.com/eregon/sequel/runs/1476292451?check_suite_focus=true#step:7:27
  I believe it would be far more useful if continue-on-error at the step level would behave more like it currently does at the job level, and mark the CI job as failed or failed-but-expected.
  The failed step itself should have a red or yellow mark, so it can be noticed without having to guess which step failed.
  GitHub Actions already knows about the outcome of every step, and that's also in API, the only missing part is showing it in a useful manner (and not exactly the same as a successful step).
• if used at the job level, then the job is marked as failed, which at least can be noticed. And the whole workflow is still marked as successful when looking in the Actions tab. But on the PR it's rather confusing, and it looks like the CI broke when it did not.
  Example: https://github.com/jeremyevans/sequel/pull/1737/checks?check_run_id=1479678900
  
  The big red Some checks were not successful sounds like something broke, but no, an error/failure there is expected so it shouldn't be marked as "CI failed". And on the wheel there is a bit of red.
  The whole area is encircled in yellow (I'm not sure why), which seems OK.
  So there I would propose to mark the failed job in yellow, and possibly use another symbol. And then to use the same color in the wheel for jobs which are continue-on-error: true. And also change the text so it would be something like All expected checks have passed in green (the wording for all checks are green is All checks have passed).
  The whole area should then be encircled in green, because it's safe to merge without looking at logs.

This from issue https://github.com/actions/toolkit/issues/399, which seems to not be getting too little attention from github while there are 100+ comments and 500+ interactions.
Somewhat related discussion: #9875

[ktomk]

This perhaps depends on perspective. I had problems to understand continue-on-error first of all (and why it is not the same as || true which I was looking for using allow-failure on Travis before), which - like in your report - was related to the differences between Github Actions and the display of failed jobs in the PR checks.

At the end of the day I understand it that continue-on-error merely controls the behaviour how the following steps in a job are executed in case of a non-zero exit status. The job is green then in any case.

Why the check is red with the PR then I never fully understood, and I only explain it to myself it is using a steps context outcome and not conclusion but that is a guess, I don't know.

From my perspective that difference (Actions vs. Checks) is the main bummer for continue-on-error.

And if it helps, for seeing if continue-on-error was in action, my suggestion is to not scan the actions' logs but the actions' summary (the thing on top), it lists it. @eregon

[eregon]

• job continue-on-error should be the same as allow-failure, I think that's clear and what everyone wants. It works fine except for the reporting in the UI (the job is not failed and is not passed, it's "expected failure" and needs to be shown as such, e.g., a yellow check mark).
• step continue-on-error seems almost the same as || true, except the extra Check failure but that's only visible on the summary, and it doesn't link to the failed step properly. This step continue-on-error if it fails needs to be shown somehow on the job, otherwise it's just too hidden, and || true would be a better choice if people never care whether the job succeeds or not. It must be visible from the PR UI whether a continue-on-error step failed or not (e.g., via a yellow check mark or so)

[ktomk]

+1 for:

• summary should link to failed step
• yellow icon on the job would be very nice

[jeking3]

This is indeed a common technique I used frequently on Travis CI and Appveyor. Here are links directly to the documentation stating one clear use case that this is for:

https://www.appveyor.com/docs/build-configuration/#allow-failing-jobs

You can configure AppVeyor to allow certain build matrix rows to fail and still continue with the rest of the build. The result of these failing jobs is not counted towards build status. This may be useful if you are experimenting with running tests on the latest version of a platform or framework, e.g. Node.js 0.11.

This is a useful technique for saying "test against the latest language level" which can change over time and can potentially fail, without violating any documented requirements of a project. In this case one does not want the sum of their checks to be "failed" or "failure", but "ok" or "passing". The yellow indicator requests related to this ask I find are not really necessary, but would not be harmful, and could be added at a later time.

https://docs.travis-ci.com/user/customizing-the-build/#jobs-that-are-allowed-to-fail

You can define jobs that are allowed to fail in the build matrix.

Allowed failures are jobs in your build matrix that are allowed to fail without causing the entire build to fail. This lets you add in experimental and preparatory builds, for example to test against runtime versions or configurations that you are not ready to officially support.

Additionally I have another use case, which is to add tests for known failures and run the tests in the future to see if they get fixed:

1. I was adding CI jobs to boostorg/mpl to improve testing.
2. One of the jobs added a new architecture and exposes a known problem already in the issues.
3. I want to leave the job in place, failing, and check it in. Then if someone fixes it, this will verify it.
4. I do not want the job to cause other pull requests or CI workflows on main branches to reflect as "failing" or "failed" because it is a known, documented issue. If this is the only failed job in a workflow, it should not fail the workflow if it were marked as allow-failure.

---

The semantics of continue-on-error are quite different from continue-on-error:

• continue-on-error allows the rest of a job or workflow to run even if something fails (i.e. do not "fail fast") but still fail the whole.
• allow-failure allows a job or step to fail, be marked as failed, but be treated as an acceptable error; any step or job with allow-failure should not contribute to the final calculation of the job or workflow. Additionally, allow-failure implies continue-on-error;
  • If this is on a step, and that step is the only step that fails, the job outcome can be failed but the conclusion should be succeeded.
  • If this is on a job, and that jobis the only job that fails, the workflow outcome can be failed but the conclusion should be succeeded.
  • Additional continuance logic should be based on the conclusion, not the outcome.

All other dependencies such as step and job dependencies should continue to work as normal.

[pboling]

It has been memory holed.

🗒️ Now that's it is missing it makes @eregon 's comment a little confusing, since what it referred to is now gone, but there is still a comment which it looks like he is replying to. I should have blanked it out instead of deleting perhaps.

As to whether or not it would have helped... it's complicated.

Github is not acting entirely with the best interests of the developer community in mind (as that would literally be illegal in the United States, where a company must always act in the interest of shareholders), and so you have to apply pressure in various ways to get companies to course correct, as that is the only way to align human (developers in this case) interests with the "interests of shareholders"... Ah, capitalism!

[hishamhm]

(@pboling Side-note: it is sort of an urban legend that companies must always act in the interest of shareholders. See here, here. The "interest of shareholders" can be used by boards to justify acts, but their acts must not always be motivated by that; ultimately the board's fiduciary duty is with the company's overall value, which may be long-term increased by actions which go against the short-term interest of shareholders — it's complicated).

[pboling]

@hishamhm Technically, you are correct. In practice, however, boards do use it as justification, especially at companies like Microsoft which are primarily vehicles for making billionaires. Boards of Directors that act explicitly in the interest of the people using their service to the detriment of their shareholders could be held liable, as it is literally their job to protect the interest of shareholders.

Also, your framing is pretty strange given that "shareholders first" has been essentially business standard practice forever, and now that it is starting to change, slightly, as noted in your last link, lawyers are having to rethink their approach to the law.

Your reading of the facts exposes your biases (as does mine).

[dhirschfeld]

Fellas, please take your discussion of the merits (or otherwise) of capitalism offline - this isn't the place. For the benefit of everyone wanting to get useful information/updates please keep the discussion here focused on the topic at hand.

[pboling]

This is the last I'll say in this issue. Github deserves to know, as do the thousands of people and projects reliant on my code, that I am leaving this platform as fast as I am able, and this issue is one of several reasons for that.

[ioquatix]

Not having a proper allow-failures (same semantics as Travis) wastes so much time and causes so much confusion in the UI. Please fix it GitHub. 100% on board with all of the OP statements.

[ioquatix]

I have just filed a bug report with GitHub Support:

Description

As discussed here, GitHub Actions is missing an "Allow Failure" mode/configuration: https://github.com/orgs/community/discussions/15452

https://github.com/actions/toolkit/issues/399 - in total over 1000 positive reactions.

What do we have to do, as a community, to communicate the importance of this feature and get movement on it?

Reproduction Steps

The desired feature is to allow some jobs to fail, and indicate it on the PR UI in a sensible way, much like TravisCI did with allow-failures: true.

An example use case is where you are creating a library of code, which can work with several different implementations. Let's say, you are targeting several implementations of node.js. Some implementations might not have all the features you need, but you still want to test them. Some builds are expected to fail when the node.js implementation doesn't support the required feature. They should not result in a net failure of the entire build matrix, because they are marked as "allowed to fail".

At some point in the future, those builds may start passing. At that point, calling out that they are passing but also allowed to fail, is useful. Because it tells you, now this particular implementation has been updated and the build is passing. You can remove it from "allowed to fail".

[mcraveiro]

"Allow failures" as an enabler of incremental improvement

(reposting here from 399 as requested by @pboling)

I have a one-person project [1] , so PR checks are not an issue for me. However, as with most in this ticket, I am need of "allow failures". Let me add my use case to the discussion as I think it is slightly different from the ones described so far, but probably a very common one.

At present in my project we do not support clang-cl (a c++ compiler for windows). The clang-cl build fails to link due to some weird link error, and I haven't had time to figure it out. If I add clang-cl to the build matrix, all my commits get marked with a red X, i.e. failed. This, even though the build was marked as experimental [2]. As a consequence, I need to check each and every commit in detail, in case some other build is broken. I could of course remove the build altogether (which is my workaround at present, see commit above). This is less than ideal because I'd rather know if a particular commit makes that build worse, or, in case I get lucky, better. So it would be nice to continue to build clang-cl in my build matrix, without impacting the overall red/green status of each commit.

In fact, by not having this feature, it makes incremental improvements to the build much harder. I now need to enable the build, deal with any rotting that may have happened since the last time it was enabled, make some changes, run the build and then disable it again. The drive-by approach I used to take - whenever I got 20 minutes here and there, do a quick experiment to see if it improved things - has now been rendered unworkable because most of those twenty minutes would be spent on overhead.

All of this to say that, for me, this is an important feature.

Links

[1] https://github.com/MASD-Project/dogen
[2] https://github.com/MASD-Project/dogen/blob/66022e278892f2fedfa44df2b56cb2741e01a28c/.github/workflows/continuous-windows.yml#L30

[marcellodesales]

I have the same problem... My slack action is failing, failing the whole pipeline, although the rest worked... So the PR is locked... At this point, it's hard to believe Github Management is way too far from the community... This feature is definitely a very important ask and would allow us to bypass situations like the one described above...

[rachelruderman]

Plus one for all of the above. I'm working on fixing flaky tests in our code that cause CI to fail, which means I'm running many, many tests and constantly checking if they succeeded or failed. We have a GitHub check for CI and another GitHub check to flag tests that failed once but succeeded on retry. The "succeeded on retry" check is optional. My work would be greatly facilitated by the optional check not causing the build to be marked a fail.

[ross-spencer]

Echoing what others are saying here, our use-case is very similar and connected to https://github.com/actions/toolkit/issues/399#issuecomment-1260017446. allow-failures would enable this.

I want to monitor Python release-candidates and forward compatibility. Just to provide a warning for the roadmap if anything fails. The red failure is a little depressing after a period of time, and not true to the intention of the workflow.

[mtaylor-evidation]

I agree this would be a nice to have. +1 the allow-failure

[johnnyshields]

allow-failure and continue-on-error are two different features which are not mutually exclusive and in fact can be combined to complement each other (see below). As these are common semantics in every other CI system on the planet, Github should implement both, and we should stop squabbling over "whether we should only eat apples or only eat oranges."

• allow-failure (could also be called ignore-failure) permits ignoring a failure of a specific job run in the matrix and still marking the overall CI task as success (green).
  For example, given I have a Ruby gem hosted on Github, and I want to check that it works against multiple Rails versions. If I test against the Rails HEAD (i.e. latest repo master branch), as it may contain unknown breaking changes, I can use allow-failure so this particular HEAD run can fail, but I still pass my CI. In this case, the UI state of the failed job should be indicated as a "Gray X" , i.e. failed, but ignored.
• continue-on-error permits a step within a job run to fail, but still move on to the next step.
  For example, given I have a step that does a report notification, which if failed I still want to run the subsequent steps, I could use continue-on-error. Here the failed step still counts a failure for the overall job run, unless I also combine it with allow-failure as per above. Here a "Yellow X" might make sense UI-wise.

[pboling]

Until Github has both of these features properly implemented (and addresses other issues I note on my profile), I will continue investing hours removing projects from Github.

[pboling]

Update: The closed issue has been moved to the appropriate, relevant, repository, and is where backlinks should point, even though it remains closed.
actions/runner#2347 (comment)

Now that this is the official, and only, forum for discussion of this issue it is worth point out that - this is a terrible forum for discussing this issue.

Github Discussions do not backlink to commits / PRs / Issues that link to them. GitHub issues, OTOH, do offer backlinks. Many, nay hundreds, nay the majority of interactions on the now-closed-issue were backlinks where people were referencing the issue from all over GitHub precisely in the context where they were having problems that would be solved by this feature.

Now any future context of that sort will be lost, and we will only get a fraction of the context, discussion, and input from the community, and devoid of the hundreds of real world examples that were available before.

Perhaps it is good to have the loudest voice control the discussion (GitHub in this case, speaking over everyone else by saying nothing at all). But if that is the case, don't elfing call it "Community Discussions". It's misleading, as it implies to most reasonable people that GitHub would be participating in the discussion. Call it "The Place To Vent While GitHub Ignores You" instead.

[mrk-han]

+1 for your use of "elfing"

And yes, these are very valid concerns. But let's try to tone down the rhetoric and keep things civil here.

[rivy]

Now that this is the official, and only, forum for discussion of this issue it is worth point out that - this is a terrible forum for discussing this issue.

Github Discussions do not backlink to commits / PRs / Issues that link to them. GitHub issues, OTOH, do offer backlinks. Many, nay hundreds, nay the majority of interactions on the now-closed-issue were backlinks where people were referencing the issue from all over GitHub precisely in the context where they were having problems that would be solved by this feature.

Unfortunately, I also agree that this is more of a "goes to quietly die" area, rather than "discussion of issue" area.
But, at least to maintain some context, here is a backlink to the larger, multi-year issue discussion thread https://github.com/actions/toolkit/issues/399.

[haltman-at]

Regardless of whether continue-on-error is changed to work like allow-failure, or whether allow-failure is introduced as a separate option and the two are kept distinct, ultimately, we need allow-failure in some form. More specfically: An allowed failure is one whose failure does not cause a failure of the level it's contained in. If one job fails, but it's an allowed failure, then that job should display red but the overall PR should display green. Etc.

...of course a ton of people have said this already, I just wanted to add my name to do what I can to underscore the importance of this...

[mansona]

Hey folks 👋 I wrote a post about this on the now infamous closed issue on this topic a while ago where I was sharing a project that I created to bring back some semblance of the "allow failures" feature that we used heavily in the Ember community when Travis was the default CI provider.

I thought I would share a link to the project here in case it helps anyone 👍

https://github.com/mainmatter/continue-on-error-comment

[akrabat]

Seeing a red X against a PR that has successfully passed CI for all shipping versions of the underlying runtime, but failed the nightly is requiring additional training of all project maintainers. I'm very surprised that GitHub closed issue actions/runner#2347 as "Not planned" with no notice of why this UX problem is considered acceptable.

[andrewakim]

Hi all 👋 - I'm a PM for Actions and looking into this issue. I'm hoping to setup 5-10 conversations with folks to dig into the problem and use cases. If you're interested in chatting, please use this Google Form to send over your details. I'll close the form and edit my post once I get enough conversations lined up. Thanks and looking forward to chatting!

Edit on Mar 3, 2023: Thanks for signing up and the responses! I have enough conversations lined up so I've closed the form. I'll reach back out to setup more conversations if needed.

[johnnyshields]

Agreed with @tomato42. This feature is so obvious and so universally implemented in other CIs, it's like we're designing a traffic light, and we need to have conservations and user stories around adding "yellow".

[ross-spencer]

+1 on all of this! plus the other comments that describe the different but largely all very complementary use-cases for this feature. I wonder if perhaps the team at GH can describe what they understand is being asked for, and what the complexities of it are for them?

[pointlessone]

it's like we're designing a traffic light, and we need to have conservations and user stories around adding "yellow".

This is a great illustration why it's being asked. Because yellow means different things to different people. Sometimes these differences come from different backgrounds (e.g. yellow means slightly different things in different countries) or different experiences that don't quite reflect reality (e.g. they never experienced any issues while crossing on yellow so they treat it "another green").

Yellow was around for so long that most people have mostly consistent and kinda OK understanding of what it is. And mostly because yellow itself went though a along process in some government committee to end up in the for you know it. That's not the case with what's being asked here. Similar features are present in different CIs but they all are slightly different. Everyone's had different experience and expect a slightly different thing. And given the size of HG's userbase adopting a random one will result in everyone else being disappointed.

[ricardoboss]

So... it's over a year later and we haven't got an update. Is this still in the planning phase?

[pboling]

@andrewakim A day on Jupiter is only ten Earth hours long. It has been almost 3 Jupiter years now. Any chance we'll get this soon?

[pllim]

All I ever wanted is a new yellow icon to show when a job that is allowed to fail failed but does not necessarily block merging. ❌ is too much and ✔️ is misleading.

[tracyhoward]

It can be a green circle with a red X in the middle for all I care, we just need a distinction.

[y-nk]

As a temporary workaround, if that can help anyone:

steps:
  - name: foobar
    run: |
      set +e
      set +o pipefail

      function wrap_command() {
        curl # here, any command which supposedly fails.
        return $?
      }

      wrap_command
      echo $?
      exit 0

I use this in a E2E test logic where we want 3 retries before calling a test "fully failed" such as:

  - name: foobar
    run: |
      set +e
      set +o pipefail

      function yarn_e2e() {
        yarn e2e $1
        return $?
      }

      for attempt in {1..3}; do
        echo '::group:: test: ${{ matrix.testFile }} ($attempt/3)'
        yarn_e2e ${{ matrix.testFile }}
        [ $? -eq 0 ] && exit 0
        echo '::endgroup::'
      done
      exit 1

[ro0NL]

i had clean/simple yaml once 😂

[adamwojt]

This is much needed. There is no way to deem workflow as "skipped" or "neutral" when one of the jobs fails but it is not so important.

[fuch1m]

+1 for this feature

[sjanc]

how hard can it be to implement this? is it being blocked on purpose or what?

[onigoetz]

If you put an AI spin to that feature there is probably a chance to get it implemented ... 🙃

[antonbaliasnikov]

+1 for this feature

[zackees]

How is this feature not supported yet?

[pboling]

This is the right question. The evident answer is that software development is not a priority for Microsoft. Possibly the most upvoted issue in the history of GitHub Actions is still dead after four years.

actions/runner#2347
actions/runner#798
https://github.com/actions/toolkit/issues/399
actions/toolkit#1034
https://github.com/orgs/community/discussions/56441
https://www.reddit.com/r/devops/comments/y2t6b9/the_feature_request_for_a_allowfailure_option_on/

It's a ridiculous foot gun.

Any updates @andrewakim ?

[LeMyst]

Product Manager recently answered: actions/runner#2347 (comment)

[pboling]

I had read that and forgotten about that, as I am currently running into this missing feature again.

@nebuk89 this is the discussion where it has been discussed most recently by a different project manager (@andrewakim), and let us know if there is a new home for this issue!

[sobomax]

[wsanchez]

I got sufficiently tired of this that I came up with a work-around. Posting it here in case others find it useful:

https://github.com/burningmantech/ranger-ims-server/pull/1347/files

What I did gets me:

• a green checkmark for the job (not great, but this is the trade-off we are making)
• a green checkmark for the workflow (the goal here)
• a comment in the PR indicating that an optional build failed (replacement for red checkmark for the job)