Exception: Request failed after too many retries

[AlexSkrypnyk]

Hi there,

I've been using Codecov for many open-source and private repos because of it's feature-rich functionality. Thank you for providing this product to the open-source community!

After the switch to v4, the reporting is being extremely unstable with PRs randomly failing for different types of projects (there has been many issues reported and multiple PRs trying to fix this).

With this config:

      - name: Upload coverage report to Codecov
        uses: codecov/codecov-action@v4.5.0
        with:
          directory: /tmp/.scaffold-coverage-html
          fail_ci_if_error: true
          verbose: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

My latest problem appear today out of nowhere:

with:
  directory: /tmp/.scaffold-coverage-html
  fail_ci_if_error: true
  verbose: true
env:
  SSH_AUTH_SOCK: /github/home/ssh-agent.sock
  CODECOV_TOKEN: ***
##[endgroup]
##[command]/usr/bin/docker exec  348886b840faec166ac2da5f6ac6cac06ed6027e90470801d417b8e5c620a0e7 sh -c "cat /etc/*release | grep ^ID"
evenName: pull_request
baseRef: drevops:develop | headRef: drevops:feature/fixed-codecov_token-in-tests-gha
evenName: pull_request
baseRef: drevops:develop | headRef: drevops:feature/fixed-codecov_token-in-tests-gha
evenName: pull_request
baseRef: drevops:develop | headRef: drevops:feature/fixed-codecov_token-in-tests-gha
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
Received SHA256SUM eb902009cb800fdefbce291c1357953f80e29382f43c7a98cce3b57d2b4d8cec  codecov
Received SHA256SUM signature -----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEJwNOf9uFDgu8LGL/gGuyiu13mGkFAma/080ACgkQgGuyiu13
mGnIlg/7BMFoXv8PE+E5DLH/lsY53dq7xRO8GtK+xfHYQgntH8EcR+Xs4Go4WAFr
iPgO7E1cVE7IfqKvf/e2tfcs1ZjE7yMqeNcYiq8EqcOOeV3JL1OiiOgQeh+hmEQU
DS/uvlST2XiR4BiX5wwWQzW+1kdy0yIy51U1HT5T0hdL6UmIzFOoCNx3iyzVRS9x
tBRtbU/PgbivQWhC9jFbm8TgzdjrX/eIT9Ei1fJex/qDZ4RKiQsYy/b7S2T683fV
jA4aJYlCLfiadscFGHFkdJl1ZR0aec9GmWRUlJiF3Ee0w+xBlu6NTWFOSx2KGM95
RvdpQB6I4YJHfWLRTQr4qE0hOtSDokOPEhs4V3TiTwFjsEUE27rwV93/NLof8mDb
QcSJD84b2MlvziMKw92rUDJiNyn72dEh2iZGRSfpbx8M5SpP9w7+aPRdpfd9g22t
6J+CyPDD9rLdmFNK1NCwNzFYgTRLy0V5M3VZCes5yhkYOnvgob75a9cEssSsqa1a
yE2h3+kc4kbUqlU4IGoYRzbCKve7ivK6WeJA8SEq6e391e1OUKoVS5t79lM1SRHc
3Y8D/HfdTX97w60IC2MkrYRkCOhigORYu6r+gXHRDggsNN7XiRZIc4euW3uhE6GA
O3HJbH83xR//mpICSBvfjyr8tONKZYRJ8CUrGauQk769rvfrkBI=
=d3Lv
-----END PGP SIGNATURE-----

gpg: directory '/github/home/.gnupg' created
gpg: keybox '/github/home/.gnupg/pubring.kbx' created
gpg: /github/home/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Fri Aug 16 22:33:49 2024 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869
==> Uploader SHASUM verified (eb902009cb800fdefbce291c1357953f80e29382f43c7a98cce3b57d2b4d8cec  codecov)
==> Running version latest
==> Running version v0.7.4
==> Running git config --global --add safe.directory /__w/scaffold/scaffold
[command]/usr/bin/git config --global --add safe.directory /__w/scaffold/scaffold
==> Running command '/__w/_actions/codecov/codecov-action/v4.5.0/dist/codecov -v create-commit'
[command]/__w/_actions/codecov/codecov-action/v4.5.0/dist/codecov -v create-commit --git-service github -C c9c882c8b52fc092155e74024c2b37e23f1f2e58 -Z
info - 2024-08-16 22:35:06,347 -- ci service found: github-actions
debug - 2024-08-16 22:35:06,349 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-08-16 22:35:06,351 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
warning - 2024-08-16 22:35:06,353 -- No config file could be found. Ignoring config.
debug - 2024-08-16 22:35:06,353 -- No codecov_yaml found
debug - 2024-08-16 22:35:06,354 -- Starting create commit process --- {"commit_sha": "c9c882c8b52fc092155e74024c2b37e23f1f2e58", "parent_sha": null, "pr": "1360", "branch": "feature/fixed-codecov_token-in-tests-gha", "slug": "drevops/scaffold", "token": "e******************", "service": "github", "enterprise_url": null}
warning - 2024-08-16 22:35:06,466 -- Response status code was 500. --- {"retry": 0}
warning - 2024-08-16 22:35:06,466 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-08-16 22:35:07,064 -- Response status code was 500. --- {"retry": 1}
warning - 2024-08-16 22:35:07,064 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-08-16 22:35:08,173 -- Response status code was 500. --- {"retry": 2}
warning - 2024-08-16 22:35:08,174 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 85, in <module>
  File "codecov_cli/main.py", line 81, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/commit.py", line 64, in create_commit
  File "codecov_cli/services/commit/__init__.py", line 28, in create_commit_logic
  File "codecov_cli/services/commit/__init__.py", line 65, in send_commit_data
  File "codecov_cli/helpers/request.py", line 82, in wrapper
Exception: Request failed after too many retries
[PYI-4715:ERROR] Failed to execute script 'main' due to unhandled exception!
##[error]Codecov: Failed to properly create commit: The process '/__w/_actions/codecov/codecov-action/v4.5.0/dist/codecov' failed with exit code 1
Post job cleanup.
##[group]Run codecov/codecov-action@v4.5.0

In my branch, where the setup is trivial, token was rotated, version was pinned to the latest available, the config of GHA is exactly as per README.md, yet the action cannot complete successfully.

The error Exception: Request failed after too many retries is kinda useless - it does not explain the reason.

Question to developers

Can this issue please be used to consider extending the logging to provide extended information on the cause of failures?

Can we also know in the log if the cause is a Codecov own service unavailability? It is understandable if the service is down (every service requires a maintenance once in a while), we just want to know in logs that it is on Codecov's side. Also, https://status.codecov.com/ shows that everything is operational, while in fact, it is not.

Question to managers and product owners ion Codecov

I apologize for the bluntness, but it’s challenging to recommend your product to my clients when these unresolved issues have persisted since the release of version 4 six months ago. The frequent PR failures are hindering our development teams so much that they are considering moving away from Codecov.

I'm hoping that the stability of this GHA can be prioritised.

[ReenigneArcher]

I'm getting this starting today as well (repeatedly). Seems to only be affecting Linux and Windows runners in my case. macOS is okay (for now). https://github.com/LizardByte/Sunshine/actions/runs/10427264605

[amotl]

Hi there,

we would like to chime in and report that we are being tripped by the same problem, starting to be observable since about an hour ago on two of our public repositories commons-codec and cratedb-toolkit, see ¹².

We guess the response is just right with Response status code was 500, so we figure the backend is currently out of service?
Thank you so much for providing this excellent service to the community, you know who you are, and good luck analyzing and fixing the problem.

Please advise if you think the problem might be on our end in any way.

With kind regards,
Andreas.

Footnotes

1. https://github.com/daq-tools/commons-codec/actions/runs/10427094602 ↩

2. https://github.com/crate/cratedb-toolkit/actions/runs/10427577777 ↩

[TomerFi]

+1

[ryanw-mobile]

Facing the same issue. The script has been unchanged, running fine for a long time. I don't believe it's my/our problem.

[thomasrockhu-codecov]

I think this may have been a result of https://status.codecov.com/incidents/1g1pznmcxr8t

[srabraham]

I started consistently seeing this problem today, and it's because codecov/codecov-action@v4 is now (newly?) connecting to ingest.codecov.io. We use a step in our runners (step-security/harden-runner) that blocks calls to unallowed endpoints, so I just need to start allowing ingest.codecov.io.

It would be nice if the failure message (Request failed after too many retries) would say what endpoint it failed to connect to.

[thomasrockhu-codecov]

@srabraham thanks for the call out there, I'll put in some work this month to get a better error message for this