Basic tests for manifests

[kimwnasptd]

We've seen in the KF 1.4 release (Manifests Testing phase), as well in the 1.4.1 patch release #2084 (comment) that testing the manifests currently takes some manual cycles.

It will help us to reduce time on testing, and thus releasing faster, by defying some basic unit and integration tests that run on each PR, to ensure that at least:

1. The manifests an be generated with kustomize
2. The underlying Pods can be created and become Ready
3. The components can work together alongside the common dependencies (Istio, Knative, K8s version)

The aim of these tests should not be to run exhaustive tests on specific components, since this falls to the WGs' territory and responsibilities. The tests should provide some very basic checks that each component is deploy-able and can work, alongside the other components.

I'd like to create a proposal for this effort since it touches a lot of parts and needs a clear definition of goals and non-goals. I think it will be a good time to also start having a slightly more formal process for more advanced features.

cc @kubeflow/wg-manifests-leads

[kimwnasptd]

Before starting with the proposal I want to evaluate where these tests should run.

Traditionally we have been running tests on Prow, by utilizing infra provided from the kubeflow/testing repo and the Optional-testing infra that is backed by AWS. So I can see the following approaches:

1. Stick with Prow, that spins up AWS clusters, and run tests using Argo (or Tekton)
2. Go with GitHub Actions to trigger the tests and use something like KinD to setup a K8s cluster

Both have pros and cons, like being closer to a production environment, ease of writing the tests, time to setup a K8s cluster etc. But before going down the comparison road I'd first want to ensure that both options are viable in the longs run.

Specifically, I'd like to have some assurance on the first approach which will help in evaluating the best long term approach. @surajkota can you provide some more insight on whether the Optional-testing will be supported in the future?

Lastly, since we are getting close to the feature freeze and manifests testing phase of the release I'd like to submit the proposal early next week, to get this effort into motion. @surajkota If we could have this information, regarding the Optional-testing infra, by end of the week it would really help us for sticking with the timeline.

[jaypipes]

@kimwnasptd Hi! I'm the engineer at AWS that originally helped @PatrickXYS and @andreyvelich with setting up the AWS credits for KubeFlow. :) I work with @surajkota at AWS on the ACK projects.

I can work with you to get additional AWS credits for the KubeFlow project granted to the AWS account that we originally used back in early 2020 (IIRC). To do so, I just need an AWS pricing calculator entry that estimates the amount of $ for the various AWS services the KubeFlow testing infrastructure would use. I will Slack you on the k8s slack to get this private conversation started, if that's OK with you?

Secondly, in the ACK projects, we have quite a bit of experience running e2e tests that create k8s clusters using KinD from within a Prow test job running on a long-running EKS cluster. This setup has been really useful to fully test the entire ACK experience from installation (via Kustomize or Helm) all through usage of the ACK controllers via the Kubernetes API. This is virtually identical to what you're describing above as needing/wanting for KubeFlow.

If you're interested, @RedbackThomson @vijtrip2 and myself would be happy to present our automation and e2e testing for ACK and how it all works for you and the KubeFlow community. Please just let us know! :)

[surajkota]

@js-ts @thesuperzapper for kubeflow/community#545, please see above ^^. Can you estimate a price for running the tests suggested in 545?

[kimwnasptd]

Hi @jaypipes @RedbackThomson @vijtrip2!

I can work with you to get additional AWS credits for the KubeFlow project granted to the AWS account that we originally used back in early 2020 (IIRC). To do so, I just need an AWS pricing calculator entry that estimates the amount of $ for the various AWS services the KubeFlow testing infrastructure would use. I will Slack you on the k8s slack to get this private conversation started, if that's OK with you?

Thank you very much, this would be great!

Secondly, in the ACK projects, we have quite a bit of experience running e2e tests that create k8s clusters using KinD from within a Prow test job running on a long-running EKS cluster. This setup has been really useful to fully test the entire ACK experience from installation (via Kustomize or Helm) all through usage of the ACK controllers via the Kubernetes API. This is virtually identical to what you're describing above as needing/wanting for KubeFlow.

That's a very interesting testing scenario! Glad to see that we can avoid re-inventing the wheel and learn from existing efforts and expertise.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to inactivity.

[kimwnasptd]

/lifecycle frozen

[juliusvonkohout]

@annajung is this still relevant for Kubeflow 1.8