[test] Avoid Rate Limit Exceeded #24931
Conversation
oliviertassinari
commented
Feb 14, 2021
•
oliviertassinari
commented
- The problem I'm trying to fix: https://app.circleci.com/pipelines/github/mui-org/material-ui/37981/workflows/933e76b5-fa9c-4b6d-a5d6-3b2d9ef1d2f7/jobs/224165
- Leads on the solution I could find:
- Use higher default pollingTimeout than 1 second karma-runner/karma-browserstack-launcher#30 "Use higher default pollingTimeout"
- https://www.browserstack.com/docs/automate/api-reference/selenium/introduction#rest-api-projects "You can make up to 1600 API requests per 5 minutes per user"
- the config https://github.com/karma-runner/karma-browserstack-launcher/blob/4547fd7660fa47721f8ec43808b369c7aa077c0b/index.js#L83
oliviertassinari
force-pushed
the
browser-stack-rate-limit
branch
from
8c91b68 to
f20a465
Compare
test/karma.conf.js
Outdated
| const MAX_REQUEST_BROWSERSTACK = 1600 / (60 * 5); | ||
| // Estimate the max number of concurrent karma builds. | ||
| // CircleCI accepts up to 80 concurrent builds, for each PR, 6 concurrent builds are used. | ||
| const MAX_KARMA_CONCURRENT_BUILD = 80 / 6; |
There was a problem hiding this comment.
The maximum amount of concurrent test_browser jobs is 80.
I'm not following how you leap from "CircleCI accepts up to 80 concurrent builds" to "for each PR, 6 concurrent builds are used". How does a PR factor in here?
If you're estimating at some point then this should be its own magic number so that we can easily adjust later. Not baked into a well defined constant.
Seems to me we don't actually care about the CircleCI concurrency (or rather just as an upper bound for verification). Instead we're just trying to handle the maximum number of parallel test_browser jobs. I'd suggest just going with "10" for now and see how it goes.
There was a problem hiding this comment.
6 comes from the number if items in:
Ok, let's use another magic number :)
There was a problem hiding this comment.
6 comes from the number if items in:
Why is the number of jobs relevant? In a review I'm not interested in the what. I can read code for myself. I need to know the reason for why did it that way because that is not obvious.
There was a problem hiding this comment.
Let's consider the following case:
- 20 open PRs at the same time, a condition that can hit the maximum build capacity.
- CircleCI can process all the jobs for the first step checkout in parallel (because 20 < 83)
- CircleCI completes all the jobs for the checkout step, more or less at the same time
- CircleCI moves to the next steps in the pipelines. This time he has to handle 6 x 20 = 120 jobs, 20 of these jobs are test_browser-1. He picks 83 jobs randomly inside these 120 jobs. test_browser-1, test_unit-1, test_static, etc. all the jobs have an independent, equal probability to be picked. So out of the 83 jobs, 83/6 are the step test_browser-1.
- We are now running ~14 test_browser-1 jobs at the same time.
=> 83/6 is the maximum parallelism Karma can hit (assuming the above simplification of the problem).
| const MAX_REQUEST_PER_SECOND_BROWSERSTACK = 1600 / (60 * 5); | ||
| // Estimate the max number of concurrent karma builds | ||
| // For each PR, 6 concurrent builds are used, only one is usng BrowserStack. | ||
| const AVERAGE_KARMA_BUILD = 1 / 6; |
There was a problem hiding this comment.
What does "AVERAGE_KARMA_BUILD" mean?
