Checks and Workflows conclusion can be different (+ Checks UI is confusing)

[justyn]

Neither annotations nor job conclusion seem to be working as expected.

If you look at this job run: https://github.com/LouisBrunner/checks-action/actions/runs/220658890

Note that the example annotations are not shown, and that tests like "test_basic_failure" and "test_basic_neutral" just show as successful.

[LouisBrunner]

Hi @justyn!

This is an issue I considered as well when building this action, should it actually fail the workflow when the conclusion is not success, action_required or neutral?

My current philosophy is to say that if you pass the result from another action, it will probably have failed itself and you are running the checks-action with if: always() anyway, so no need to explicitly fail. The issue of annotations not showing up is linked to GitHub itself, but I understand your frustration.

What behavior did you expect? Maybe there is a way to make both methods compatible (extra flag)?

[justyn]

Perhaps I misunderstood the purpose of this action?

I had thought it was to expose results of tests (or other operations) in the Github checks results.

For example, I expected test_basic_failure to show a X next to it in the Action tab, rather than a tick. And test_basic_neutral to show a neutral icon, etc.

From your response I have a feeling I may have misread the purpose?

[LouisBrunner]

The action is just a wrapper around the GitHub Checks API, so it adds the requested checks with the given information. That is different from the result of a workflow, which is shown as the cross or a tick on the commit/PR (even though I think a workflow shouldn't be valid if a check has failed but alas).

If you check my showcase PR you can see that all the checks, and the ones which use failing conclusions will appear with a cross (or another non-tick icon). It also shows on the Action page, but not on all of the Action pages, I don't really know why?

In summary, the action doesn't currently do anything about your workflow passing/failing as it focuses solely on the Checks API. However I had similar thoughts while writing it and the fact that you expected it to work that way makes me think it might be a good addition?

[justyn]

Ah okay the link shows what I was expecting (more or less).

In the run I linked to in the issue, that isn't happening, which is what prompted my confusion. Also when I ran it in my own workflow, I see the same thing (that is, not what is expected) which I why I raised the issue.

Even in the run you linked to, I see duplication, so that there is a test_basic_failure with a tick and then a Test Basic Failure with the expected cross which is somewhat confusing.

[LouisBrunner]

Yeah, it is really confusing. I don't personally understand why GitHub shows the Checks sometimes but not other times. It always seems to show on the PR page but not necessarily on the Actions page...

I agree that the duplication is confusing, test_basic_failure is the workflow and Test Basic Failure is the resulting check. You could also have a multitude of checks who can pass or not for a single workflow...

Maybe I am calling the API in a weird way and cause the Checks to disappear sometimes? In any case, I don't really know how to improve these issues...

On the other hand, I can make the state of the workflow and check closer by failing the checks explicitly in the workflow if that's something you would find helpful.

Sorry I can't help more, I was pretty frustrated with it myself when I put it together... :/

[justyn]

No worries, thanks for your help, and thanks for publishing this action!