GitHub Actions - Ratelimiting poping up suddenly after not having affected me before.

[FxllenCode]

Description
Over the last week, I have noticed GitHub is consistently rate limiting me whereas this has never happened before. I have not made any changes to the actions workflow. There is not at all anything that should be causing this. You can find my repository that is affected here:

https://github.com/Status-Plus/StatusPlus

This is critical for me, it has completely broken my system as I rely on actions, it is simply disappointing that I am being rate limited now, considering this "rate limit" is clearly undocumented.

Area for Triage:
Deployment/Release

Question, Bug, or Feature?:
Bug.

Virtual environments affected
All.

Expected behavior
GitHub actions should run as it does normally, like it did before.

Actual behavior
I am being rate limited to the point where my systems do not work at all now:

You can see where this starting coming in place:

Here is the error message:

From what I am reading, it says that the issue is related to missing authentication, however I don't see where this may have changed. I will look into that however.

Repro steps
I'm really not sure how I can reproduce this, it just started randomly happening to me. Here is the repo:

https://github.com/Status-Plus/StatusPlus

Thank you.

[Darleev]

Hello @FxllenCode,
I found a well-documented page regarding API rate limits you can find it below
Rate limit page
Also, we had a very similar request in the past, which possibly can be helpful to resolve the current issue:
#602
Additionally, I would like to provide you StackOverflow solution on how it is possible to increase the number of unauth/auth requests:
https://stackoverflow.com/questions/13394077/is-there-a-way-to-increase-the-api-rate-limit-or-to-bypass-it-altogether-for-git

Could you please check all information provided and let us know if we need to investigate the issue further?
We are looking forward to your reply.

[FxllenCode]

Hello @Darleev

I've checked everything, (and to be fair I am no expert in GitHub actions,) it seems that the VM being used is Linux, so MacOS should not be the problem. I am not too sure why it just started happening recently, I did not make the requests happen any faster. (It says that all 1000 of the rate limit is being used by the actual number should be around 100...)

I do believe this is probably a bug relating to some update in the past couple of weeks as I can't find any reason I would suddenly start getting rate-limited.

Thanks!

[Darleev]

@FxllenCode Could you please add a rate limit checker to the workflow as per documentation?

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/rate_limit

It should return a number of remaining API connections for authenticated users.

[FxllenCode]

@Darleev unless I misunderstood, this already is here:

I know for a fact there have not been 1000 calls of GitHub actions here-- thus why I think this is a problem related to the way GitHub actions is checking rate limits.

[maxim-lobanov]

@FxllenCode , on the quick look, it definitely looks like your actions exhaust GH Rate limits.
graphs.yml is run every 10 minutes and based on the logs it does ~549 requests as minimum.
It means ~5490 requests per hours that is more than 5k limitation. In additional, you have other actions that do API requests too.

I suggest temporary disabling graph.yml scheduling to see if other actions will work as expected.

[FxllenCode]

@maxim-lobanov gotcha. I will attempt this and see how it goes, thanks.

[maxim-lobanov]

@FxllenCode , Btw, looks like Rate limit is 1000 requests only if you use system token:

When using the built-in GITHUB_TOKEN in GitHub Actions, the rate limit is 1,000 requests per hour per repository.

If you replace it to your own PAT, rate limit will be 5k requests per hour but you should understand security concern whether you trust actions that you use (upptime/uptime-monitor and others) or you probably can try generating PAT with minimal permissions.

[FxllenCode]

@FxllenCode , Btw, looks like Rate limit is 1000 requests only if you use system token:

When using the built-in GITHUB_TOKEN in GitHub Actions, the rate limit is 1,000 requests per hour per repository.

If you replace it to your own PAT, rate limit will be 5k requests per hour but you should understand security concern whether you trust actions that you use (upptime/uptime-monitor and others) or you probably can try generating PAT with minimal permissions.

That's odd. I thought I was using PAT. I'll look into that, as well.

EDIT: I was correct, I am using PAT! That might be why I am just now getting rate-limit issues. Is it possible my PAT has expired?

[maxim-lobanov]

@FxllenCode , I am not sure that you use own PAT.
In YAML file, you are referencing repository (system) token: https://github.com/Status-Plus/StatusPlus/blob/master/.github/workflows/graphs.yml#L32
According to the documentation: https://docs.github.com/en/actions/reference/authentication-in-a-workflow

If you would like to use own PAT, it should be placed in repository secrets to some variable like GH_TOKEN and reference this variable in YAML. As far as I know, secrets.GITHUB_TOKEN is referencing system token.

[maxim-lobanov]

Update:
I think if PAT is expired - it would fail on the first request (I am not sure honestly).
Also, if your own PAT is located in variable with name GITHUB_TOKEN, I suggest renaming it to be different from system token variable. I am not sure how this conflict is actually resolved.

[FxllenCode]

The PAT is named GH_PAT, so I am not too sure if that is a problem.

As for upptime, I am just following the documentation here:

https://upptime.js.org/docs/get-started#add-repository-secrets

I just updated my PAT token as according to GitHub it was last used 2 weeks ago, when the problem started. For now, I'll assume the token got expired. (No idea why however).

The only reason I assume this isn't an upptime problem and is a GitHub problem is because I have had zero issues for the last 3 months until about 2 weeks ago.

It's important to note that PAT may not be expired because a couple of requests go through, not many however.

I have also slowed down Graphs CI significantly to once every 24 hours so that may also solve the issue.

I just wonder why this issue only popped up recently.

[maxim-lobanov]

@FxllenCode , GH_PAT is not used in your workflow.
It is not enough to just add GH_PAT in repository secrets, you should add it to YAML file.

- name: Generate graphs
  uses: upptime/uptime-monitor@v1.26.1
    with:
      command: "graphs"
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      GH_PAT: ${{ secrets.GH_PAT }}

[FxllenCode]

I'll bring this up with the creator of upptime. Thanks for your help. I'll take a look and see if it fixes on my end, will close this for now as I assume you are correct about this. Thanks again!

[maxim-lobanov]

Cool,
Btw, looks like official upptime example use correct way to pass own PAT: https://github.com/upptime/upptime/blob/master/.github/workflows/graphs.yml#L32
GITHUB_TOKEN: ${{ secrets.GH_PAT }} - map GH_PAT to GITHUB_TOKEN.

[FxllenCode]

Cool,
Btw, looks like official upptime example use correct way to pass own PAT: https://github.com/upptime/upptime/blob/master/.github/workflows/graphs.yml#L32
GITHUB_TOKEN: ${{ secrets.GH_PAT }} - map GH_PAT to GITHUB_TOKEN.

It looks like the official upptime repo is incorrectly calling GH_PAT as GITHUB_TOKEN, probably from the latest update. Unfortunately this fix has to be done on their end as workflows are run and updated on that side. I'll have to manually rename GH_PAT to GITHUB_TOKEN for now as the code itself is looking for a secret called that:

Thanks!

[catthehacker]

GITHUB_TOKEN is automatic token secret that gives read/write access to repo and packages - https://docs.github.com/en/actions/reference/authentication-in-a-workflow#using-the-github_token-in-a-workflow
What you need to do is change secrets.GITHUB_TOKEN to secrets.GH_PAT in workflow and add your own PAT in secrets of that repo with name GH_PAT.
upptime repo is correctly calling everything. They use env.GITHUB_TOKEN for auth and it's user responsibility to assign valid auth token to that environment variable.