Flake? E2E Test failed with GCP quota exceeded

[fasaxc]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

CI tests for #48469 failed with a quota error:

I0708 13:01:19.525] Creating new network: e2e-38580
W0708 13:02:07.576] ERROR: (gcloud.compute.networks.create) Could not fetch resource:
W0708 13:02:07.577]  - Quota 'SUBNETWORKS' exceeded.  Limit: 100.0

Full logs:

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/48469/pull-kubernetes-kubemark-e2e-gce/38580/build-log.txt

What you expected to happen:

Clean test run.

How to reproduce it (as minimally and precisely as possible):

I don't know wnough about the test infrastructure to know.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): Unknown (test infra)
• Cloud provider or hardware configuration**: GCP
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[k8s-github-robot]

@fasaxc There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

[fasaxc]

@kubernetes/sig-testing-misc

[k8s-ci-robot]

@fasaxc: Reiterating the mentions to trigger a notification:
@kubernetes/sig-testing-misc.

In response to this:

@kubernetes/sig-testing-misc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[0xmichalis]

/assign @krzyzacy

[0xmichalis]

@krzyzacy IIUC, your new tool, boskos, is destined for fixing these kinds of failures, right?

[krzyzacy]

no, that should be janitor, and seems something is leaking badly from the test

[krzyzacy]

I'll check the project, @MrHohn IMO it does not even run test, any suspicious?

[MrHohn]

Quick update: Seems like the auto-network includes more regions than before so we hit quota limit when too many PR jobs are running together.

[krzyzacy]

sent a quota bump, will need a better design in the future.

[krzyzacy]

/close