Document how to properly setup Codecov Github Action for OpenSource repositories

[ST-DDT]

Describe the bug

PRs from external contributors fail CI.

The PR is happening in a forked repo. Using tokenless upload.
Tokenless has reached GitHub rate limit.

Environment (please complete the following information):

• CodeCov GitHub-Action v4.1.0
• https://github.com/faker-js/faker/blob/a672d277b883a445a852f9b6db18818104e8de7e/.github/workflows/ci.yml#L212-L216

To Reproduce

Steps to reproduce the behavior:

1. Go to an OpenSource project with CodeCov GitHub Action
2. Open a PR as external contributor
3. The CI will very likely fail (~10 CI failures, sometimes multiple in a row)

External contributors have a near 100% chance at failing. Internal contributors have a near 0% chance at failing.

Expected behavior

The CI is expected to work.
Or alternatively the guide describing how to setup the action should provide hints on how to deal with the error.

Screenshots

info - 2024-03-12 15:43:46,354 -- ci service found: github-actions
info - 2024-03-12 15:43:46,690 -- The PR is happening in a forked repo. Using tokenless upload.
info - 2024-03-12 15:43:47,076 -- Process Commit creating complete
error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}
Error: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/54bcd8715eee62d40e33596ef5e8f0f48dbbccab/dist/codecov' failed with exit code 1

Additional context

Created on request here:

• "Unable to locate build via Github Actions API" for the public repository #126 (comment)

Affected CI pipelines:

• https://github.com/faker-js/faker/actions/runs/8251702824/job/22569395448?pr=2740
• https://github.com/faker-js/faker/actions/runs/8251662161/job/22569253564?pr=2724

[kdarkhan]

It might be hard to find the workaround I came up with from the linked issue, adding it here if that becomes useful.

I ended up using 2 workflow approach which seems to work pretty stable with token based auth for forked repos.

This workflow uploads coverage report to Github artifacts and this workflow uses that report and the token to upload it to codecov.

[adamjstewart]

Any updates on this? We're seeing a 100% failure rate on most recent PRs. Is the open source community expected to transition to a different coverage provider, or are there future plans to once again support forks?

[rohan-at-sentry]

@ST-DDT thanks for raising.

For context (which you probably already realized from the screenshot - but may benefit others), v4 allows tokenless uploads for forks ; enabling open-source contributors to upload coverage.

Because this happens without a token, Codecov uploads from forks upload using Codecov's own token - you can see how quickly that can get consumed leading to a failure. That is most likely what is happening when you see

error - 2024-03-12 15:43:47,076 -- Commit creating failed: {"detail":"Tokenless has reached GitHub rate limit. Please upload using a token: https://docs.codecov.com/docs/adding-the-codecov-token. Expected available in 1455 seconds."}

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

Also - I'm curious if the workaround here worked for you

[ST-DDT]

In effect - uploading from a fork should be (from a reliability standpoint) no different from before v4 was released. Is it possible for you to confirm if the rate is worse since Feb of this year?

I think the rate remained the same.
It just happened a lot at the time of reporting (and for an extended period of time).
Previously it was an undefimed error, now it is a specific error.

Also - I'm curious if the workaround here worked for you

No, we are currently in a feature freeze and thus have very few external contributions.
We generally have a low need for code coverage and thus consider removing it instead due to the additional work required.

[adamjstewart]

I tried to use the above workaround. Didn't try the full thing with hundreds of lines of JavaScript code, but it was too complicated to get working with the usual upload/download artifact actions. TorchGeo gets 99% external contributions, so we're very keen on getting this working.

[rohan-at-sentry]

Hey folks cc @ST-DDT @adamjstewart

We're exploring the best path forward for a fix here. Here's the current thinking - codecov/engineering-team#1574

We'd expect this to be ~ 4 weeks of work.. I'll report back here once this is complete. We're shooting for something that doesn't require more config.

thank you for your patience.

[rohan-at-sentry]

Meanwhile, we've better documented the tokenless flow in our docs . In general, there shouldn't be a need to implement workarounds to pass the token to forks that can then use it to upload to Codecov.

[LebedevRI]

It would be nice to expand

(and the latest version of the Codecov Action)

with an information about the actual first version that supports that behaviour.

[adamjstewart]

Could also replace:

there may be cases when uploading to Codecov via tokenless fails

with:

there may be cases when uploading to Codecov via tokenless works

as the former is much more common (at the moment) 😏

But seriously though, thanks for all of your hard work to improve the situation. I know how difficult it is when GitHub refuses to increase the rate limit for a token and security is involved. I'm sure you're drowning in bug reports at the moment. Looking forward to a working solution, and happy to help test on our repo!

[rohan-at-sentry]

It would be nice to expand

(and the latest version of the Codecov Action)

with an information about the actual first version that supports that behaviour.

Agreed, updated

[rohan-at-sentry]

Could also replace:

there may be cases when uploading to Codecov via tokenless fails

with:

there may be cases when uploading to Codecov via tokenless works

as the former is much more common (at the moment) 😏

But seriously though, thanks for all of your hard work to improve the situation. I know how difficult it is when GitHub refuses to increase the rate limit for a token and security is involved. I'm sure you're drowning in bug reports at the moment. Looking forward to a working solution, and happy to help test on our repo!

Thanks for your patience as we work through this. <3

[jotak]

Hi folks,
I was wondering if it would be possible to use codecov on PRs (via pull_request GH event) only for posting comment and without uploading the report? Report uploads being done via another workflow, on push events, which doesn't have the same impact on secrets visibility. If that's technically doable, I guess it would be beneficial not only to fix the failures we're currently seeing, but also in terms of lowering the traffic & attempts between codecov and github api ?

[adamjstewart]

That is the workaround discussed in #126. See #126 (comment) for why that's currently way too difficult to get working properly. Of course, if there was a documented solution or simplification of this that worked, that would be fine. But a solution that allowed PR coverage to be uploaded without a secret and upstream repo coverage to be only uploaded with a secret would also be fine with me.

[jotak]

@adamjstewart I was thinking something slightly different - but maybe this is naive as I don't really know how codecov works under the hood :

• Reports get only uploaded when PRs are merged (ie push event). There's no report uploaded during the PR opened lifetime
• When PR is opened (pull_request event), the report is generated, and diff is performed locally against the base, comment is posted, but nothing being uploaded
  I guess that's not possible to do with the current state of the codecov-action, but my question is, would it be a good solution to implement. Or perhaps it's not technically doable :-) What I'm not sure is if the action has the ability by itself to run the diff against the base locally.

[adamjstewart]

A comment would be better than nothing, but the ability to view the diff and missing coverage on the codecov website would be better. I don't think it would be a security issue to upload PR coverage without a secret, as long as a secret is still required for protected branches like main.

[kdarkhan]

Based on previous comments it seems Codecov is working on a solution to this. I don't know how they are planning to resolve this but I believe something like the workaround I used in #301 (comment) can be packaged as reusable actions. The workaround works pretty well but is verbose to implement.

They could define 2 GitHub actions:

• coverage report generator (does not upload but saves to GH artifacts)
• coverage uploader (uploads from GH artifacts to Codecov)

If done properly then users could just use 2 actions without extra verbosity of JS scripting.