Specify check suite when creating a checkrun

[jwgmeligmeyling]

When creating a check run through the API, a check suite is newly created unless a check suite for the requesting application has already been made, in which case that check suite seems to be reused.

This leads to issues however: due to Github Actions workflows, and each Workflow Run (workflow/event) having its own check suite, multiple check suites may exist for a given commit SHA. As a result, annotations generated in a Github action during a step executed in the “pull request” event, end up in the “workflow” for the "push’ event.

Through hoops, it is possible to find the check run ID during a Github Action: one can look up the Workflow Run by the run_id and then extract the Check Suite ID from the checkSuiteUrl. However, there is no way to specify the Check Suite ID when creating a new check run (its ignored in the request) and neither there seems to be any dedicated endpoint for creating check runs for a particular check suite.

It would be great if it were possible to more easily find the corresponding Check Suite for a Workflow Run, and if it were possible to create Check Runs for this particular Check Suite rather than have one picked “randomly” for you.

See also the issue thread jwgmeligmeyling/pmd-github-action#4 and proof of concept code in jwgmeligmeyling/pmd-github-action#3

[francisfuzz]

jwgmeligmeyling:

It would be great if it were possible to more easily find the corresponding Check Suite for a Workflow Run, and if it were possible to create Check Runs for this particular Check Suite rather than have one picked “randomly” for you.

@jwgmeligmeyling - Welcome back to the Support Community and thanks for this feedback! 🙇‍♂️

After reviewing your particular case, I’d like to confirm here that it is not currently possible to create a check run associated with a particular check suite with the GitHub Actions-provided GITHUB_TOKEN. I acknowledge that it is something that would be valuable given your use case.

We’re always working to improve GitHub and the GitHub Support Community, and we consider every suggestion we receive. Would you mind submitting this through our official product feedback form so that our product team can track your request? That’s the best place to share requests like these in consideration for future iterations of GitHub features, including Actions.

[carlspring]

@francisfuzz ,

Ping? Please, see the comments below.

[ruben-hoenle-fnt]

@francisfuzz
Ping?! Please, see the comments below!

[samus-aran]

Hi @ruben-hoenle-fnt 👋

Thanks for sharing and we appreciate your patience. You can check our public roadmap to see the kinds of work we are focusing on for the coming year. Smaller fixes won’t be listed, but are often included when work is happening adjacent to it.

While it has been a while since we have updated on this thread, the absence of an update is simply that - there has been no update because there is so far nothing to report. That doesn't mean it isn't being worked on - it might simply be that we're unable to share details about work that isn't ready for release.

As a GitHub user, the most you can do is to share your feedback on this Discussion space with the correct tags (include examples), and trust that we are always working to make GitHub as awesome as possible. We can't guarantee that every bit of feedback sees action, but we are paying attention.

[jwgmeligmeyling]

Thanks for your response! I’ve submitted the product feedback form.

[bewuethr]

Is it really useful to copy-paste ChatGPT answers here ("last knowledge update in September 2021")?

[queenofcorgis]

@Akashh2151 While we encourage non-AI answers to questions in community, we also love experimenting and using new technologies. We ask if you do use ChatGPT or similar programs, please disclose what software and model version number you used to generate the response you used in your reply in order to promote transparency in the community.

ChatGPT answers in a tone that sounds like a tech support professional and while we support trying out this software, it is important to verify that they are solutions before posting otherwise it may mislead users into thinking that they are receiving official responses when they are not.

Per ChatGPT's website, the program does have limitations like:

• "May occasionally generate incorrect information"
• "May occasionally produce harmful instructions"
• "Limited knowledge of world and events after 2021"

[jwgmeligmeyling]

So instead of reporting this junk as spam we encourage it and ask for the version number? This whole thread is getting more insane by the day.

[EnricoMi]

Is this product feedback submitted by @jwgmeligmeyling publicly visible? What is the best way to track progress of this issue?

[jwgmeligmeyling]

No I don’t think its publicly available. I didn’t receive any response as well. I’ve filed another support ticket, maybe that gets things going.

[francisfuzz]

@EnricoMi - Great question! Because product feedback submitted through this form isn’t publicly visible, it’s not possible to track its progress. However, it may be worth sharing that our team announced the GitHub public roadmap to give everyone more transparency into what each of our teams are planning and to help everyone understand our product vision. It’s publicly viewable if you’re interested in following what our team is working on next, though it doesn’t currently show any feedback submitted from that form. I hope this helps!

[jonico]

It might be worth trying out to use the token of a separate GitHub App when creating the check run. Here is an action that retrieves a temporary token from a GitHub App created just for this purpose: https://github.com/peter-murray/workflow-application-token-action

[NasAmin]

Yes I have the same problem.
When you have multiple workflows triggered on a pull_request event and one of the workflows is creating check runs using the Create Check API, the check runs end up randomly in one of the workflows.

I have also submitted a suggestion on the feedback form.

[dobegor]

@francisfuzz is there any update on this issue? I wasn’t able to find it on the roadmap you’ve mentioned.

[eblot]

I was expecting to use several runners to perform distinct, unrelated checks, but the results appear in a randomly picked workflow tab. Is there any known workaround for now? Thanks.

(note: it sounds weird to have this thread flagged as “solved” :-))

[francisfuzz]

@dobegor - I don’t have any updates to share on my part nor am I able to provide any in the future. The public roadmap is still the best place to keep up with upcoming features.

[lukash]

Hello @francisfuzz, this seems to be a “minor” bug from the roadmap PoV, and is unlikely to ever be listed there. Hence this response from you is not really useful at all, as we don’t even know whether the issue is still being tracked by Github internally or it just “fell through the cracks”.

It’s a shame especially because at least on the surface this seems like it should be an easy fix to actually use the Check Suite ID. For my use case it’s causing this: Report check item appears under the wrong workflow name · Issue #39 · ScaCap/action-surefire-report · GitHub and again, it’s a shame, because that action allows reasonable and very simple test result reporting via workflow annotations, which is clear and easy to find/read even by outside contributors. There are very few options with regards to test result reporting that don’t require outside resources!

I have submitted another feedback form for this, hopefully it can get some more attention. Thank you.

[thegecko]

This issue makes it difficult to locate checks for specific workflows and is serious enough for us to consider moving away from GH actions 😦

[inket]

This is a BREAKING issue for GitHub Actions, and it’s honestly mindblowing that it’s come down to “submit feedback and get no answer”. We have zero acknowledgment that our feedback was even seen by anyone working on GitHub Actions.

It’s especially sad since we all know this would be a very easy fix. You can literally put an intern on it they’ll get it done in less than a day.

[radiomix]

It reminds me of the old 🪟 behavior It's not a bug. It's a feature

[jwgmeligmeyling]

Sad to see this was still not communicated to the appropriate folks. It is a trivial fix. A dozen people have requested this feature.

And annotations are practically useless as it is.

Someone made a stinky API decision and successfully managed to sweep it under the carpet. Of course we want to be able to specify to which check run annotations should be added, rather than have one picked randomly for us! It’s just common sense.

[MostefaKamalLala]

This is seriously needed by my team. We moved to github actions and we are even paying for the service. We cannot have test results, annotations spread all over the place randomly. When will we get an update on the API?

[AlexPerathoner]

Same problem on my side. Definitely a problem, or "missing feature", if you prefer.

[leviplj]

Is GH going to fix this or should we give up on the checks?

[jwgmeligmeyling]

You should give up. Ive waited for years.

Talked to my GH Enterprise rep about it, they push on sarif uploads trough a Github Advanced Security upsell. Mind you, it can process a dozen files at most before crashing on you. The annotation API actually works better.

[JustinOhms]

Well this seems to explain why my check runs keep keep ending up associate with my push action and not the action driven by the PR. I can't believe this is an issue. If you cannot control what the check run is actually associated with it's next to useless.

[sarnold]

I can't believe I'm still reading this...

[carlspring]

@francisfuzz , @maxprit ,

Can we raise the severity of this? It's not just a desired feature, it's a total misimplementation of how it should work. As you can see, quite a few other developers feel this way and are affected by it.

What do we need to do in order to get the attention of the Github developers/managers? Is there a issue tracker where this can be reported? How do escalations work?

[steve-todorov]

We are also experiencing this issue and it is quite disappointing to see such a rookie mistake being made on such an important feature.
To be honest this feature makes absolutely no sense to even exist or be available to the general public, because practically it doesn't work.

Nobody ever has just a hello world project - people have 1, 2 sometimes 3 workflows that deal with the different phases / stages.
I really hope this gets more attention.

[dev-hanz-ops]

There is a workaround. It's possible to publish the check-run in a dedicated, separate check-suite by using a token for a GitHub app, instead of the regular GITHUB_TOKEN.
As example, we use this action:

• create a GitHub App (permissions e.g. Read and write access to actions, checks, code, discussions, and pull requests - possibly too broad for your case)
• create a private key for it and store it in your GH Actions secrets
• install the app to your org / repo

•       - id: app_token
          uses: tibdex/github-app-token@b62528385c34dbc9f38e5f4225ac829252d1ea92
          if: always()
          env:
            OPENSSL_CONF: /dev/null  # https://github.com/tibdex/github-app-token/issues/54
          with:
            app_id: your-app-id
            private_key: ${{ secrets.PRIVATE_KEY_YOUR_APP }}
  
        - name: Publish Test Results
          uses: EnricoMi/publish-unit-test-result-action@v2
          if: always()
          with:
            # when using the regular GITHUB_TOKEN, the check-run created by this step will be assigned to a
            # random workflow in the GH UI. to prevent this, we can force the check-run to be created in a separate
            # check-suite, which is created automatically if we use the credentials of a GitHub App
            github_token: ${{ steps.app_token.outputs.token }}
  

update: your workflow should create a comment like the default one using the github-actions user, as otherwise the action will create one comment per workflow run, instead of updating the first:

      - name: create test-results comment if not exists
        if:
          always() && (github.event_name == 'workflow_dispatch' || steps.changes.outputs.src == 'true')
        env:
          GH_ENTERPRISE_TOKEN: ${{ secrets.GITHUB_TOKEN }}    # choose one
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}               # choose one
          GH_HOST: your-enterprise-github.com                 # leave blank if using github.com
        run: |
          set -o xtrace
          short_ref=$(IFS='/' read -r -a REF <<< "$GITHUB_REF" && echo ${REF[2]})
          test_result_comment=$(gh pr view $short_ref --comments \
            --json comments \
            --jq '.comments[]
              | select(.author.login == "github-actions")
              | select(.body | startswith("## Test Results"))')
          [[ -z "$test_result_comment" ]] && gh pr comment $short_ref --body $'## Test Results\nResults for commit ' || true

result

It comes at the cost of not having the published message as part of the GH Actions run - this would otherwise be randomly assigned.

[austinpray-mixpanel]

We already have GitHub actions workflows that publish checks under a github app. The feedback we have been getting in a large repo is that the UX is bad because it disconnects the job the generates the checks from the checks. Also you have to be super careful multiple github actions (pull_request, push, schedule) writing to the same check suite stomping all over each other.

Conceptually it's a github action that generates checks. Why not be able to colocate the checks with the thing that caused them?

[steve-todorov]

I completely agree. We noticed it as well, but there is simply no "going around it". 😞

@francisfuzz will get annoyed at some point if we ping him enough times. :)

[carlspring]

@steve-todorov

@francisfuzz will get annoyed at some point if we ping him enough times. :)

Unless he's muted notifications from this thread... :)

@maxprit,

Could anyone at Github comment on this? It's obviously an issue for quite a few people.

[EnricoMi]

Maybe raising individual issues at Github Support may get their attention.

[jwgmeligmeyling]

I am not sure why the sudden influx on this thread. I appreciate y'alls trust in getting this resolved, but after pushing this issue for three years now I can only recommend to put your time elsewhere

[EnricoMi]

Github support's response is:

Using the Checks API to generate check_runs at runtime from an Actions workflow is not a supported use scenario.

The API is meant to be called from a Github app, but not from Githubs very own Github app called Github Actions.

The proper use in Github Actions is to create your own Github app, install that app in your repos, get an installation token and use that with the Checks API (see https://github.com/orgs/community/discussions/24616#discussioncomment-5607870 above).

Very user-friendly.

[carlspring]

This discussion sounds like a front row seat at a stand-up comedy show! 😀

[mutantcornholio]

Just to clear something out: if you have a single workflow that is triggered by serveral different webhook events, you can't have it resulting in a single status on a commit?

An example:

• an action is triggered on a pull_request event
• then somebody reviews a PR (triggering a pull_request_review event)

So the action can't create a commit status in first case, which would be updated in the second?

[samrees]

I spent all day on this before I found this thread, trying to merge my status between those two events.

[alexmaurizio]

It is now May 2024 and this trivially fixable problem still plagues the whole GitHub Actions ecosystem.

I'm definitely baffled.

[bfollonikea]

Encountered this issue recently. Sad to see this thread was open in 2022 and the problem still exists.

[mashedkeyboard]

Just encountered this myself and spent a solid ten minutes staring at the code, the workflow run, and the GitHub support form, before finding this discussion... this seems bizarre! Adding a voice to ask for this to be fixed please.

[eXorus]

Same for me, I had to redo all my code to account for this bug.

[rashadatjou]

+1 experiencing this issue with this awesome eslint action helper ataylorme/eslint-annotate-action#35

[diegopalat]

Any news on this topic?