Make the kubeflow-m2m-oidc-configurator a CronJob

[kromanow94]

Which issue is resolved by this Pull Request:
Resolves #2646

Description of your changes:
Changing the Job to CronJob improves the robustness of the setup in case if the JWKS will change or the user accidentally overwrote the requestauthentication.

Checklist:

• Tested on kind and on vcluster.

[kromanow94]

@juliusvonkohout or @kimwnasptd can we restart the tests? Both of them failed because of a non-related issue:

timed out waiting for the condition on pods/kubeflow-m2m-oidc-configurator-28537425-s8kzm
timed out waiting for the condition on pods/activator-bd5fdc585-rrnqf
timed out waiting for the condition on pods/autoscaler-5655dd9df5-4knpj
timed out waiting for the condition on pods/controller-5447f77dc5-ljx5r
timed out waiting for the condition on pods/domain-mapping-757799d898-knf69
timed out waiting for the condition on pods/domainmapping-webhook-5d875ccb7d-z2qjv
timed out waiting for the condition on pods/net-istio-controller-5f89595bcb-dv7h2
timed out waiting for the condition on pods/net-istio-webhook-dc448cfc4-rws5f
timed out waiting for the condition on pods/webhook-578c5cf66f-25sf9
timed out waiting for the condition on pods/coredns-5dd5756b68-hpg77
timed out waiting for the condition on pods/coredns-5dd5756b68-vv66m
timed out waiting for the condition on pods/etcd-kind-control-plane
timed out waiting for the condition on pods/kindnet-9l886
timed out waiting for the condition on pods/kindnet-pftsz
timed out waiting for the condition on pods/kindnet-z5qpl
timed out waiting for the condition on pods/kube-apiserver-kind-control-plane
timed out waiting for the condition on pods/kube-controller-manager-kind-control-plane
timed out waiting for the condition on pods/kube-proxy-64vj7
timed out waiting for the condition on pods/kube-proxy-vk4lr
timed out waiting for the condition on pods/kube-proxy-xwm8d
timed out waiting for the condition on pods/kube-scheduler-kind-control-plane
timed out waiting for the condition on pods/local-path-provisioner-7577fdbbfb-7zv5k
timed out waiting for the condition on pods/oauth2-proxy-86d8c97455-hvjl8
timed out waiting for the condition on pods/oauth2-proxy-86d8c97455-z9vjw
Error: Process completed with exit code 1.

[juliusvonkohout]

@KRomanov, i restarted the tests. If they fail again we might have to increase the timeouts in this PR.

[kromanow94]

@juliusvonkohout this is super weird. I limited the CronJob with concurrencyPolicy: Forbid. I don't know if this should be handled with increasing the timeout or by increaseing the resources for the CICD Jobs... I can also try to split the installation steps to limit how many pods are created at the same time...

[juliusvonkohout]

I restarted the tests. yeah our CICD is a bit problematic at the moment. If we can specify more resources in this public repository yes, otherwise we have to increase the timeouts. https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

[kromanow94]

@juliusvonkohout maybe the issue is with CICD resource sharing? If the memory and cpu is shared between multiple workflows, it may be problematic. I see one of the failing tests completed with success. Can you restart the last test workflow?

Also, is this something I could do myself, for example with the github bot with commands in comment?

[juliusvonkohout]

SHould we not go with every 5 minutes instead of every minute?

[kromanow94]

I can change to every 5 minutes. There is also configuration for not adding more jobs until the last one is completed and from the latest log from cicd workflows shows that there is no more than 1 job created at a time.

[juliusvonkohout]

are you sure that script.sh is idempotent?

[kromanow94]

Huh, well it doesn't verify if the JWKS is present and after all is always performing the patch so this might be an improvement. I think the JWKS value should be also compared and only patched if different.

[kromanow94]

I made changes so the script will first check for the JWKS present in RequestAuthentication and only patch if not equal to the desired JWKS.

[juliusvonkohout]

@juliusvonkohout maybe the issue is with CICD resource sharing? If the memory and cpu is shared between multiple workflows, it may be problematic. I see one of the failing tests completed with success. Can you restart the last test workflow?

Also, is this something I could do myself, for example with the github bot with commands in comment?

I did restart and it failed again. In the KFP repository that was possible with /retest or /retest-failed or so. Probably something i can investigate in the next weeks when i am less busy.

[kromanow94]

@juliusvonkohout maybe we could add verbosity to the logs in CICD GH Workflows? We currently know that the pods aren't ready but what is the actual reason? DockerHub pull rate limits? Not enough resources? Failing Pod?

[juliusvonkohout]

@juliusvonkohout maybe we could add verbosity to the logs in CICD GH Workflows? We currently know that the pods aren't ready but what is the actual reason? DockerHub pull rate limits? Not enough resources? Failing Pod?

Yes, lets do that in a separate PR with @codablock as well.

[juliusvonkohout]

The tests in #2696 were successful so i reran the test and hope that the CICD is happy now. If not please rebase the PR against the master branch.

[juliusvonkohout]

https://github.com/kubeflow/manifests/actions/runs/8891109875 here is the successful test.

[juliusvonkohout]

So we need a rebase and step by step debugging with minimal changes.

[juliusvonkohout]

/hold

[juliusvonkohout]

/retest

[kromanow94]

@diegolovison this is the PR we've discussed on the Manifests WG Call.

[juliusvonkohout]

@kimwnasptd @rimolive please also review

[juliusvonkohout]

/lgtm

[juliusvonkohout]

/hold

[kromanow94]

Huh, the CICD failed again with:

Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'x-powered-by': 'Express', 'www-authenticate': '***"http://10.96.103.33:8888/pipeline/apis/v2beta1/experiments?filter=%7B%22predicates%22%3A+%5B%7B%22operation%22%3A+1%2C+%22key%22%3A+%22display_name%22%2C+%22stringValue%22%3A+%22m2m-test%22%7D%5D%7D&namespace=kubeflow-user-example-com", error="invalid_token"', 'content-length': '22', 'content-type': 'text/plain', 'date': 'Thu, 13 Jun 2024 15:35:11 GMT', 'server': 'istio-envoy', 'x-envoy-upstream-service-time': '3'})

https://github.com/kubeflow/manifests/actions/runs/9502404627/job/26190294755?pr=2667

But, only 1/3 m2m cicd workflows failed... I'll have another look and try to pinpoint why this is happening. I guess this should be rather a small change.

[diegolovison]

Going to wait for the feedback

[kromanow94]

I made changes to the script so it will patch and then verify if the patch with jwks persisted. If it's not persisted, the Pod will finish with failure. The CronJob is configured to restart failed Job Pod 3 times. If this also fails, the ./tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh will also fail.

@diegolovison , @juliusvonkohout , @kimwnasptd , please review.

[juliusvonkohout]

@diegolovison can you test now?

@rimolive we have to merge this for rc.2

[juliusvonkohout]

/lgtm
/approve

There was no feedback for a week and we need it in the next RC @rimolive

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

/unhold