[feat] add gke release testing

[shaneutt]

What this PR does / why we need it:

This adds testing of conformant, production grade Kubernetes clusters as a pre-requisite for any release.

Which issue this PR fixes

Supports #1095

Special notes for your reviewer:

This is only one iteration, there are several related and follow-up items:

• Webhook integration/e2e tests don't run on GKE (disabled on all but kind) #1613
• Legacy Ingress for KIC 2.0 #1614
• Knative ingress tests don't work on GKE #1617
• Release Testing: Latest Kubernetes #1616

Do Not Merge:

There's some git history cleanup that the author needs to perform before merging, please let the author merge this when its ready.

[mflendrich]

Is it a good idea to review before git history cleanup? These tend to introduce bugs.

[shaneutt]

Is it a good idea to review before git history cleanup? These tend to introduce bugs.

In this case, it's literally just dropping the release workflow trigger on every PR (so that we can verify that the tests are passing before merge, but not leave them on for all prs).

[rainest]

For posterity, example run at https://github.com/Kong/kubernetes-ingress-controller/runs/3203219168?check_suite_focus=true, which looks fine.

It seems like we'd maybe just want to skip the bulk ingress test on GKE to avoid modifying it (I assume to deal with GKE resource quotas), as that's to stress the controller code internals and shouldn't have any difference compatibility-wise from the single-ingress tests. Doesn't look like it really runs slower though (3m random traditional run versus 4m for this), so 🤷

Do we have example failure runs or other documentation of the issues with the Knative and AC tests? The TODOs don't mention them and it seems like we'd definitely want to follow up on those.

[shaneutt]

For posterity, example run at https://github.com/Kong/kubernetes-ingress-controller/runs/3203219168?check_suite_focus=true, which looks fine.

It seems like we'd maybe just want to skip the bulk ingress test on GKE to avoid modifying it (I assume to deal with GKE resource quotas), as that's to stress the controller code internals and shouldn't have any difference compatibility-wise from the single-ingress tests. Doesn't look like it really runs slower though (3m random traditional run versus 4m for this), so shrug

Do we have example failure runs or other documentation of the issues with the Knative and AC tests? The TODOs don't mention them and it seems like we'd definitely want to follow up on those.

The changes to the bulk ingress test were just modifying what were semi-arbitrary values: reducing the staggered deployment in half does still leaves the test ultimately verifying what it originally intended to verify. At this point if we want to get deep into the characteristics of performance at a specific scale, I would suggest we treat that as follow up in our upcoming performance testing features: #1197

[shaneutt]

Things are looking good. I'm going to remove the test trigger and mark this for a rebase merge once the remaining CI workflows settle. 👍