no comment on PRs when using v5

[ReenigneArcher]

Most of the dependabot PRs in my org updating from v4 to v5 do not have a codecov comment

• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/GSMS#237
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/libdisplaydevice#124
• chore(deps): bump codecov/codecov-action from 4 to 5 LizardByte/Moonlight-XboxOG#13
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/Sunshine#3399
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/shared-web#70
• chore(deps): bump codecov/codecov-action from 4 to 5 LizardByte/pacman-repo-builder#34
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/setup-release-action#137
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/tray#40
• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/support-bot#370

But one does:

• build(deps): bump codecov/codecov-action from 4 to 5 LizardByte/homebrew-release-action#106

[jwillemsen]

Same happening for me for jwillemsen/daikin_onecta#350

[jrfnl]

I can confirm as well, though we have comments turned off, but the status check is also not reporting in.

What I expect to see:

And the statuses just never show up.

On repos where branch protection is being used and the statuses are required for merging, this now means merging PRs would be blocked:

I think this issue needs the "Priority: high" label.

[dashdanw]

Just to mention, experiencing the same issues here. Have verified that the conditions are identical between instances which use @v4 and report status badge + comments, where upgrading to @v5 no longer creates either.

[yanyongyu]

Same issue for me and i check the codecov upload files in the website page, there is nothing uploaded. (action log says upload complete)

[thomasrockhu-codecov]

Got it, sorry this is taking so long. I'm going to try to fix tomorrow morning.

[dashdanw]

@thomasrockhu-codecov thanks for your work

[thomasrockhu-codecov]

@ReenigneArcher @jwillemsen @jrfnl @dashdanw @yanyongyu @wbaldoumas

ok, I released 5.0.4, would you be able to see if that fixes the issue?

[wbaldoumas]

@thomasrockhu-codecov upgraded to v5.0.4, but it doesn't look like it resolved things: wbaldoumas/markdown-colorcode#174

[ReenigneArcher]

@thomasrockhu-codecov unfortunately, there is still no PR comment with v5.0.4 LizardByte/libdisplaydevice#124

[jrfnl]

@thomasrockhu-codecov Restarted the builds for both above mentioned repos, no luck.

[dashdanw]

same here @jrfnl @thomasrockhu-codecov

[thomasrockhu-codecov]

@wbaldoumas @jwillemsen @jrfnl @dashdanw @yanyongyu, ok, I think there was an issue on our backend with a different race condition. Thanks for the patience and all the retries, and would you mind seeing if it works for you now?

[jrfnl]

@thomasrockhu-codecov Nope. Things are worse instead of better.

The builds aren't even finishing anymore, but failing with the below error instead:

error - 2024-11-21 00:55:15,401 -- Commit creating failed: {"message":"Token required because branch is protected"}
==> Failed to create-commit
    Exiting...
Error: Process completed with exit code 1.

Keep in mind that the workflows for these repo have not been changed (other than changing the v4 to v5), so since v4, they both already provide a codecov token.

[jwillemsen]

Looks to be a step back, I also now get Upload failed: {"message":"Token required because branch is protected"}

[jwillemsen]

Also the github check does show green, shouldn't this kind of error result in the failure of the github action?

[thomasrockhu-codecov]

@jrfnl I saw on the Dependabot PRs that it seems they don't have access to the CODECOV_TOKEN. Can you check that that token has been added also as a Dependabot secret?

@jwillemsen still digging into this PR

[thomasrockhu-codecov]

@jwillemsen actually, can you also check that Dependabot has access to the secret also?

[jwillemsen]

@jwillemsen actually, can you also check that Dependabot has access to the secret also?

I added the codecov token as dependabot secret but also change on codecov the setting that a token is not required anymore, now I have a new comment on my PR, thanks for all work

[thomasrockhu-codecov]

@jwillemsen are you talking about this setting?

If so, that seems like a bug, I'll have to prod the product team about that

[jwillemsen]

Yes, also set that to not required

[thomasrockhu-codecov]

@jwillemsen ok, I know that your issue is resolved, but I'm going to track that issue here: #1693

[yanyongyu]

I also get the error: Upload failed: {"message":"Token required because branch is protected"}

actions running on pull_request event do not has access to the secret. from codecov docs: https://docs.codecov.com/docs/codecov-tokens#uploading-without-a-token

it seems i need to change the token authentication option.

[thomasrockhu-codecov]

@yanyongyu do you have a link to CI you can share?

[HarelM]

My experience is different in maplibre-gl-js.
I've enabled tokenless upload but still didn't get a PR comment.
I enabled the codecov app on the repo and the comments returned, but when I disable the codecov app the comments stopped.
The codecov app is creating noise in the CI checks section, so I prefer to remove it if possible, the PR comment is sufficient for my need.
Here's a PR where upload did not fail but there was no PR comment:
maplibre/maplibre-gl-js#5074
Run details:
https://github.com/maplibre/maplibre-gl-js/actions/runs/11923312553/job/33231425376

[yanyongyu]

do you have a link to CI you can share?

sure. it's nonebot/nonebot2#3119

[thomasrockhu-codecov]

@yanyongyu I'm a little confused, it looks like you are getting a PR comment on that link.

[thomasrockhu-codecov]

@HarelM got it, that looks like an older version of the Action which probably means that the fixes we put in haven't hit. Are you still experiencing this issue?

[yanyongyu]

@yanyongyu I'm a little confused, it looks like you are getting a PR comment on that link.

i just turn the token authentication option to not required, rerun the action, the upload and comment is success now.

[HarelM]

@thomasrockhu-codecov I just opened a PR to test if I get a comment if the app is disabled, and it didn't oost a comment:
See here:

• Test codecoverage after removing the repo from org level maplibre/maplibre-gl-js#5093

So I don't think it uses an older version.
You can let us know when to disable the app if you want to do some tests.
In any case we'll enable it again until this is solved.

[thomasrockhu-codecov]

@HarelM just curious what you meant by disable the app, digging into this

[thomasrockhu-codecov]

@HarelM if you mean the Codecov app, yeah it actually looks like something is going wrong with permissions. Can you uninstall the Codecov app and re-install it? Aside from that, everything looks fine to me.

[HarelM]

When I uninstall it, the comment does not appear. This wasn't the case for version 4.
I do not want it installed, or need it (it marks build as failing, which I'm sure I can configure, but lack the motivation to do so),
but I do need and want the coverage comment.

[thomasrockhu-codecov]

@HarelM hmmm, I don't think you will be able to get notification from Codecov without the app installed. Can you show me a screenshot specifically of what you mean by what you are install/uninstalling so that there's not a miscommunication?

And as far marking a build as failing, what do you mean by that? Is there a specific Codecov status check that fails or something else?

[dashdanw]

@wbaldoumas @jwillemsen @jrfnl @dashdanw @yanyongyu, ok, I think there was an issue on our backend with a different race condition. Thanks for the patience and all the retries, and would you mind seeing if it works for you now?

Ran once which failed, received an alert on the CodeCov link generated in the output when uploading that I needed to create a new commit, did so by rebasing via main and it worked!

Thank you

[HarelM]

@thomasrockhu-codecov the following is the configuration in the org level, I mean install/uninstall when adding or removing the relevant repository from the list (in the following image it is installed on maplibre-gl-js, and other two repos):

Here's an example where the app was working, but the build is marked as "red" (red x at the top), while I consider this build a success (if the app report was not part of the "CI runs" it would have been green)
maplibre/maplibre-gl-js#5092

I've been in contact with the codecov team in the past, so you have my mail and personal info, if anything is not clear with the experience I'm trying to explain please reach out.

The bottom line from my point of view is that version 4 worked as desired and there's a problem with version 5. If this is intentional and you want to nudge people to install the app this is totally acceptable, but it's not written anywhere as far I can see, so I'm expecting the version 4 behavior to remain from my developer experience.