Inject Windows custom binaries for use in PRs and running against Kubernetes CI

[jsturtevant]

What type of PR is this?
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1276

Special notes for your reviewer:

is based on top of #1119. Will remove draft status once that PR merges.

requires kubernetes-sigs/cluster-api#4662 to run against k/k main branch

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

Adds ability to run Windows Kubernetes e2e tests against CI and PR artifacts

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[jsturtevant]

/assign @chewong

this will require the updates in #1389 to run against the main branch of k/k

[chewong]

any plans to migrate away from dockerhub?

[jsturtevant]

yes, once we move to HostProcess containers we will build these images just like the Linux Kube-proxy image.

[chewong]

/retest

[chewong]

Few nits. Otherwise LGTM (other than the kustomize issue)

[jsturtevant]

pull-cluster-api-provider-azure-e2e-windows is failing due to not having linux nodes. For upstream tests we don't need the Linux nodes but we do want one around for our capz specific tests. working on a fix

Opened kubernetes/test-infra#22485 for validating upstream but is blocked on kube-test viper change being worked on in #1421

[jsturtevant]

/test pull-cluster-api-provider-azure-upstream-v1alpha4-windows

[jsturtevant]

/hold
until #1421 merged with the kubetest updates so we can run kubernetes/test-infra#22485

[jsturtevant]

/test pull-cluster-api-provider-azure-upstream-v1alpha4-windows
/test pull-cluster-api-provider-azure-windows-upstream-with-ci-artifacts
/retest

[marosset]

/retest

[CecileRobertMichon]

Error: Unexpected non-nil/non-zero extra argument at index 1:
<*errors.errorString>: &errors.errorString{s:"system machine pools not ready"}

@alexeldeib is this a known flake? I don't think I've seen that one before

[jsturtevant]

looks like e2e-exp is passing again. I will run the conformance jobs and the PR job

/test pull-cluster-api-provider-azure-upstream-v1alpha4-windows
/test pull-cluster-api-provider-azure-windows-upstream-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-v1alpha4

[jsturtevant]

The tests passed last round but I had a bad commit message, I've rebased

/test pull-cluster-api-provider-azure-upstream-v1alpha4-windows
/test pull-cluster-api-provider-azure-windows-upstream-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-v1alpha4

[jsturtevant]

@CecileRobertMichon tests have passed several times in a row now :-) no changes other than the rebases.

[CecileRobertMichon]

/lgtm
/approve

Thanks @jsturtevant! I look forward to running these in CI

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CecileRobertMichon, devigned

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CecileRobertMichon,devigned]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[CecileRobertMichon]

/hold

@jsturtevant one of the tests is failing and it's not reporting failure https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-provider-azure#capz-pull-upstream-v1alpha4-main-windows

I assume this is because of https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/1546

[jsturtevant]

yes, this type of issue is occurring regularly on Linux as well. I believe it is a flake but can't really track it down till it we get more data. That test isn't a part of this set of changes anyways. It is part of the changes merged in #1119

I don't think adding code fixes to the PR would valuable anyways. Once we get the CI running regularly I can figure out if it is a flake or something else.

[CecileRobertMichon]

fair, let's get this merged

/hold cancel