Using sarif in GHA?

[lazka]

I' tried to set up security scanning with sarif as describe in the docs: https://github.com/msys2/msys2-autobuild/blob/1bc0a28e352f10c2da458fafa4a919c4dcf77ff3/.github/workflows/test.yml#L42-L64

I then created a test PR which introduces a warning, but it's not showing anything: msys2/msys2-autobuild#103 (correction: it showed the warning on the separate code scanning page for that PR, but there was no indication of any warnings on the PR page itself)

Does anyone know what I'm missing?

[woodruffw]

Hmm, that setup looks right -- could you try running without --format=sarif and see if the normal output produces the finding there?

Also, you do you see anything here? https://github.com/msys2/msys2-autobuild/security/code-scanning?query=pr%3A102+is%3Aopen

(I can't see that last link since I'm not a maintainer, but findings should appear there as well.)

[lazka]

Hmm, that setup looks right -- could you try running without --format=sarif and see if the normal output produces the finding there?

Yeah, it fails here: msys2/msys2-autobuild#104

Also, you do you see anything here? https://github.com/msys2/msys2-autobuild/security/code-scanning?query=pr%3A102+is%3Aopen

It no longer does now that I closed the PR, but it did show the alert there before ("warning: artipacked"). But I don't want to check that page manually for every PR to see if something is wrong. There is a high chance I would miss warnings that way.

(I can't see that last link since I'm not a maintainer, but findings should appear there as well.)

Note that I've never used the code scanning feature of GitHub, so my expectations might be off.

[woodruffw]

It no longer does now that I closed the PR, but it did show the alert there before ("warning: artipacked"). But I don't want to check that page manually for every PR to see if something is wrong. There is a high chance I would miss warnings that way.

Yeah, agreed -- at least on my repos the findings do show up on PRs, although GitHub seems to do some weird heuristics to determine when to show them. In particular, they might not show findings until you un-draft the PR, so you might want to try that 🙂

To prevent yourself from missing findings, I recommend using a ruleset to force the CI to fail if zizmor has any unresolved findings. For example, this is what I have on zizmor itself: the PR isn't allowed to merge if zizmor produces any findings of any severity:

[lazka]

huh, thanks, I'll try if that changes anything

[lazka]

I've created a rule targeting all branches:

Then I created a new PR: msys2/msys2-autobuild#105

There is a warning on the code scanning page for that PR (https://github.com/msys2/msys2-autobuild/security/code-scanning?query=pr%3A105+tool%3Azizmor+is%3Aopen+):

but the PR shows nothing and lets me merge it:

(I'll just give up on this now... Thanks for your help!)

[woodruffw]

Huh, very strange. Yeah, I don't know why that is, sorry. I'll leave this open so that someone else can hopefully chime in!

Thanks for your detailed responses!