Prow: drop max azure CI clusters to 5 #5074
Conversation
After we updated azure disks: openshift/installer#2367 We require 34 cores per cluster. This means (at a current limit of 200vCPUs) we can only safely support 5 running jobs at a time. Signed-off-by: Yu Qi Zhang <jerzhang@redhat.com>
openshift-ci-robot
added
the
size/XS
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: droslean, yuqi-zhang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
@yuqi-zhang: Updated the
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This is not a fix. We have WAY more than that number of jobs. |
… Random Azure regions Shard over four regions to increase our capacity, because Azure allows auto-bumps for vCPU limits up to 200 in each region, but it's a longer process to get the limits raised beyond that in a single region. This sets us up for more lease-restriction relaxing after 78ade84 (Prow: drop max azure CI clusters to 5, 2019-09-18, openshift#5074) and 594930f (Prow: increase max Azure CI clusters to 20, 2019-09-18, openshift#5084). We may want per-region quotas, but for the moment I'm just hoping that our random choices are even enough that an Azure-wide quota is sufficient. And conveniently, $RANDOM runs from 0 through 32767 [1], so our modulo 4 value is evenly weighted :). The change to Bash is because Bash supports $RANDOM [1], but POSIX does not [2]. We already use RANDOM in openshift-installer-master-presubmits.yaml since ca464ed (openshift/installer: add IaaS-agnostic E2E test, 2019-06-20, openshift#4148). Ideally we'd be loading the set of region choices (and possibly weights) from some shared location somewhat like ca464ed has things. And we'd be reporting the chosen region in a structured way for convenient monitoring. But the plan is to break up these templates into more composable chunks soon anyway, so I'm ok if we aren't all that DRY in the short term. [1]: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#index-RANDOM [2]: https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_05_03
After we updated azure disks: openshift/installer#2367
We require 34 cores per cluster. This means (at a current limit of
200vCPUs) we can only safely support 5 running jobs at a time.
Signed-off-by: Yu Qi Zhang jerzhang@redhat.com