Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub rate limit triggered when syncing a lot of repositories #49

Closed
grahammcculloch opened this issue May 26, 2021 · 9 comments
Closed
Labels
bug Something isn't working released

Comments

@grahammcculloch
Copy link
Contributor

🐞 Describe the bug

When executing the repo-file-sync-action in my workflow, it shows as successfully having completed but the logs show multiple instances of the following error:

Error: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.

Each time this error is reported, the PR fails to be created. For example:

Repository Info
Slug		: jellyfish-queue
Owner		: product-os
Https Url	: https://github.com/product-os/jellyfish-queue
Branch		: default
	
Locally syncing file(s) between source and target repository
Pushing changes to target repository
Creating new PR
Error: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.
Error: HttpError: You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.

No PR is created on the jellyfish-queue repo.

📚 To Reproduce

Create a sync.yml file that should result in PRs being created on a lot (about 20) of repos.

💡 Expected behavior

No errors logged, all expected PRs successfully created.

🖼️ Screenshots

N/A

⚙️ Environment

  • Action version: 1.7.0

📋 Additional context

I'm not sure if this means the GitHub user associated with GH_PAT has been rate-limited or whether repo-file-sync-action or the workflow itself has been rate-limited?

Either way, I wonder if it would help to add a RATE_LIMIT option or similar that could be used to throttle the action's requests to the GitHub API to mitigate this error?

@grahammcculloch grahammcculloch added the bug Something isn't working label May 26, 2021
@BetaHuhn
Copy link
Owner

Hey @grahammcculloch, thanks for creating the Issue!

I'm not sure if this means the GitHub user associated with GH_PAT has been rate-limited or whether repo-file-sync-action or the workflow itself has been rate-limited?

The GitHub PAT was probably rate-limited.

I wonder if it would help to add a RATE_LIMIT option or similar that could be used to throttle the action's requests to the GitHub API to mitigate this error?

What would the RATE_LIMIT option do? Would it be a fixed delay which we would wait for between requests or repositories?

We could also automatically throttle the number of requests when we encounter the rate limit. I will have to take a closer look at GitHub's Rate Limits, maybe they return a value indicating when we can make a request again.

I don't think we would need the RATE_LIMIT option if we can detect the rate limit automatically.

What do you think?

@BetaHuhn BetaHuhn added the help wanted Extra attention is needed label May 26, 2021
@BetaHuhn BetaHuhn changed the title Rate-limit hit when executing action Action triggers GH rate limit when syncing a lot of repositories May 26, 2021
@BetaHuhn BetaHuhn changed the title Action triggers GH rate limit when syncing a lot of repositories GitHub rate limit triggered when syncing a lot of repositories May 26, 2021
@grahammcculloch
Copy link
Contributor Author

@BetaHuhn yeah, I agree with your analysis. The GitHub docs on 'dealing with abuse rate limits' are quite useful with their suggestions.

I guess you could handle repos serially instead of concurrently.

Or you could enqueue all GitHub requests through a queue that throttles them to wait at least a second between requests.

I guess this is an edge-case. But presumably, others will have similar issues in due course.

@BetaHuhn
Copy link
Owner

Thanks for taking a closer look @grahammcculloch!

It might be an edge case, but as you mentioned more people are probably going to run into it. This definitely needs to be fixed.

I will take a look at your proposed solutions and will implement one of them as soon as I have time this weekend.

@grahammcculloch
Copy link
Contributor Author

Awesome - appreciate it @BetaHuhn. Do ping me if you have something you want me to test.

@BetaHuhn
Copy link
Owner

I implemented a fix on the develop branch.

The action now uses plugin-throttling.js to throttle API requests based on the mentioned recommended request throttling best practices.

I also refactored (ec2d589) the GitHub part of the action to simplify the code, but that shouldn't have any effects on the action itself.

@grahammcculloch you can try it out by using the action like this:

uses: BetaHuhn/repo-file-sync-action@develop

I don't have that many repos available right now to test this myself, so please report back if you are still hitting any rate limits.

@BetaHuhn BetaHuhn added waiting for response Waiting for a response from the author and removed help wanted Extra attention is needed labels May 29, 2021
@grahammcculloch
Copy link
Contributor Author

grahammcculloch commented May 31, 2021

@BetaHuhn awesome - I will test this out today or tomorrow and get back to you with the results!

@grahammcculloch
Copy link
Contributor Author

@BetaHuhn no problems running with the develop branch - although I can't prove that it hit the rate limit and backed off :-)

@BetaHuhn
Copy link
Owner

BetaHuhn commented May 31, 2021

Great, I will merge it into master and release a new version!

@BetaHuhnBot
Copy link
Collaborator

🎉 This issue has been resolved in version 1.7.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@BetaHuhn BetaHuhn removed the waiting for response Waiting for a response from the author label May 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working released
Projects
None yet
Development

No branches or pull requests

3 participants