GitHub rate limit triggered when syncing a lot of repositories

[grahammcculloch]

🐞 Describe the bug

When executing the repo-file-sync-action in my workflow, it shows as successfully having completed but the logs show multiple instances of the following error:

Error: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.

Each time this error is reported, the PR fails to be created. For example:

Repository Info
Slug		: jellyfish-queue
Owner		: product-os
Https Url	: https://github.com/product-os/jellyfish-queue
Branch		: default
	
Locally syncing file(s) between source and target repository
Pushing changes to target repository
Creating new PR
Error: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.
Error: HttpError: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.

No PR is created on the jellyfish-queue repo.

📚 To Reproduce

Create a sync.yml file that should result in PRs being created on a lot (about 20) of repos.

💡 Expected behavior

No errors logged, all expected PRs successfully created.

🖼️ Screenshots

N/A

⚙️ Environment

• Action version: 1.7.0

📋 Additional context

I'm not sure if this means the GitHub user associated with GH_PAT has been rate-limited or whether repo-file-sync-action or the workflow itself has been rate-limited?

Either way, I wonder if it would help to add a RATE_LIMIT option or similar that could be used to throttle the action's requests to the GitHub API to mitigate this error?

[BetaHuhn]

Hey @grahammcculloch, thanks for creating the Issue!

I'm not sure if this means the GitHub user associated with GH_PAT has been rate-limited or whether repo-file-sync-action or the workflow itself has been rate-limited?

The GitHub PAT was probably rate-limited.

I wonder if it would help to add a RATE_LIMIT option or similar that could be used to throttle the action's requests to the GitHub API to mitigate this error?

What would the RATE_LIMIT option do? Would it be a fixed delay which we would wait for between requests or repositories?

We could also automatically throttle the number of requests when we encounter the rate limit. I will have to take a closer look at GitHub's Rate Limits, maybe they return a value indicating when we can make a request again.

I don't think we would need the RATE_LIMIT option if we can detect the rate limit automatically.

What do you think?

[grahammcculloch]

@BetaHuhn yeah, I agree with your analysis. The GitHub docs on 'dealing with abuse rate limits' are quite useful with their suggestions.

I guess you could handle repos serially instead of concurrently.

Or you could enqueue all GitHub requests through a queue that throttles them to wait at least a second between requests.

I guess this is an edge-case. But presumably, others will have similar issues in due course.

[BetaHuhn]

Thanks for taking a closer look @grahammcculloch!

It might be an edge case, but as you mentioned more people are probably going to run into it. This definitely needs to be fixed.

I will take a look at your proposed solutions and will implement one of them as soon as I have time this weekend.

[grahammcculloch]

Awesome - appreciate it @BetaHuhn. Do ping me if you have something you want me to test.

[BetaHuhn]

I implemented a fix on the develop branch.

The action now uses plugin-throttling.js to throttle API requests based on the mentioned recommended request throttling best practices.

I also refactored (ec2d589) the GitHub part of the action to simplify the code, but that shouldn't have any effects on the action itself.

@grahammcculloch you can try it out by using the action like this:

uses: BetaHuhn/repo-file-sync-action@develop

I don't have that many repos available right now to test this myself, so please report back if you are still hitting any rate limits.

[grahammcculloch]

@BetaHuhn awesome - I will test this out today or tomorrow and get back to you with the results!

[grahammcculloch]

@BetaHuhn no problems running with the develop branch - although I can't prove that it hit the rate limit and backed off :-)

[BetaHuhn]

Great, I will merge it into master and release a new version!

[BetaHuhnBot]

🎉 This issue has been resolved in version 1.7.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀