Surfacing GH Rate Limit Errors

[aj-codecov]

Problem to Solve
When users hit a rate limit error in GH they don't necessarily know that it is happening and their Codecov experience is degraded because of it - we need to surface the errors and possible solutions when this issue is encountered.

Proposed Solution
When Codecov encounters a GitHub rate limit error

1. Update PR comment to show a message - "Unable to calculate coverage because GitHub rate limit has been exceeded. Please ensure the Codecov App (https://github.com/apps/codecov) is set up on this repo "
2. On the UI --> Commits tab, show a message in the "Coverage" and "Patch" columns - "Unable to calculate coverage because GitHub rate limit has been exceeded. Please ensure the Codecov App (https://github.com/apps/codecov) is set up on this repo"
3. On the pulls tab, show a message in the "coverage on Head", "patch" and "change from base" columns - "Unable to calculate coverage because GitHub rate limit has been exceeded. Please ensure the Codecov App (https://github.com/apps/codecov) is set up on this repo"

WIP: designs

Platform

Preview Give feedback

1. [Rate Limits] Technical writeup for error collection #1418
   +0
   
2. [GH Rate Limit] - Shared: Move Github Auth Adapter to Shared #1968
   task +1
   
3. [GH Rate Limit] - Add Functions to Detect Rate Limit Redis keys #1977
   task +1
   
4. [GH Rate Limit] - Worker: Use Newly Imported Auth Library from Shared #1969
   task +1
   
5. [GH Rate Limit] - Ensure API/Worker env vars are the same for bots logic #2017
   +0
   
6. [GH Rate Limit] - Bugs #2027
   task +1
   
7. [GH Rate Limit] - Worker: Create FN to Rate Limit Calls in Shared #2164
   +0
   

Applications

Preview Give feedback

1. [API] Add a check GH rate limit resolver #2103
   +0
   
2. [Gazebo] Add GH rate limit messaging on pull and commit detail pages #2107
   +0
   
3. [Gazebo] Add rate limit messaging to the resync button on the owner page #2110
   +0
   

[codecovdesign]

• Are there any potential fixes users/orgs are able to do?
• How about the checks in PR comments - how are these affected?

WIP: designs

[rohan-at-sentry]

@codecovdesign here are the answers

The only path forward is to ensure orgs have the Github bot set and are uploading using the token. Technically they can retry, but the rate limit is typically set for an hour (worst case they have to wait 60 min) so it's probably not worth letting them know to retry.

With shelter rolling out gradually, there's a scenario where we would end up showing the comment after rate is available again for everyone affected. But that is probably medium term.

PR comments would almost never show up when rate limited today
Checks - can sometimes show up if the admin has set up Codecov check as required.

I think for a first pass, showing the message in the UI and check should be sufficient. (the design is a good approximation of how we want the experience to be)

[rohan-at-sentry]

Eventually, with shelter available for 100% of orgs , I wonder if we will have an opportunity to say something like "We had trouble receiving this report due to GH rate limit, we will get it eventually"

[codecovdesign]

related reported issue of ux, in scenario not logged in:

Is the “No Files covered…” expected here? I see the number 3 on “Files changed” and expected to see them below. I am on a rather slow internet connection at the moment (on a plane), so maybe that’s it?

no feedback appears to user and/or incorrect that "no files to show" vs likely rate limit issue

[codecovdesign]

review with @adrian-codecov 3/7

first principles

• what are rate limits?
  • as it relates to GH, our system release on themfor a lot that we do. namely and importantly, when there is upload we're given commit id/hash, but not the whole story. when uploads begin processing 1) upload ingestion, 2) processing, 3) finalizing report aggregation (all in worker). 99% calling gh to fill in blank of pr author, pr upon an upload (to confirm); in this moment we there may be a limiting factor reported from GH.
    • there are a couple places where we detect the rate limit, but not a clear guideline we follow. that is to say any endpoint that calls GH, the first step would be there is a widespread detection. If we don't know that is happening, first step is to investigate these touch points and add trigger points to show the rate limit.
• what causes rate limit
  • https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/rate-limits-for-github-apps
  • there is a different rate limit for public vs private repositories, public is smaller (unsure exact numbers)
  • using a token also optimizes for larger rate limits
  • github app allows for substantially more calls
• what triggers the rate limit events
  • pr comment vs app, how align messaging
  • how to trigger removal of rate limitation
• what are the potential fixes for users
  • github app is necessary, for further investigation

action items

here are a couple places where we detect the rate limit, but not a clear guidline we follow. that is to say any endpoint that calls GH, the first step would be there is a widespread detection. If we don't know that is happening, first step is to investigate these touch points and add trigger points to show the rate limit.

further follow up on any loose ends around GitHub app requirements, are there alternatives to consider

something too look at is rest/graphql relationships

[codecovdesign]

sync w/ @adrian-codecov @RulaKhaled

• general overview - WIP investigation
• scope of work to general rate limits in pr comment, but also there is syncing (could be later)
  • differentiation: this could happen in other place (such as upload area, app, repo page)
• looking at alternative way of notifying system of rate errors, at the least the web app should be ok (unaffected by rate limit)
  • clarify: we know when rate limits are happening; notably when we are limited and looking to notify user in PR comment we have to send another request
  • implications here are that the pr comment may not show the error comment

[Adal3n3]

Making a note here: When codecov-comments bot hits a rate limit, instead of telling human go to the doc, we should tell them "Please ensure the Codecov App is set up on this repo" or "Please install Codecov App"

[codecovdesign]

@codecovdesign todo: add repo rate limit scenario