Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: Scripts for e2e tests #2128

Merged
merged 10 commits into from
Feb 10, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
4 changes: 4 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[flake8]
docstring_convention = google
exclude = assets,__init__.py,__pycache__
ignore = D100,D103,D104,D107,W503
11 changes: 8 additions & 3 deletions .github/workflows/manifests_unittests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,14 @@ jobs:
- name: Check out repo
uses: actions/checkout@v2

- name: Unit Test
- name: Install Kustomize
working-directory: ./tests/e2e
run: |
cd tests
make test
curl -Lo ./kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
chmod +x ./kustomize
sudo mv kustomize /usr/local/bin

- name: Unit Test
run: |
kustomize build example

2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
.vscode
.DS_Store

**/*.pyc

# Swap
[._]*.s[a-v][a-z]
!*.svg # comment out if you don't need vector files
Expand Down
7 changes: 7 additions & 0 deletions contrib/kserve/upstream/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kserve.yaml
# For KF 1.5 we are including both KFServing and KServe. Thus we install the
# standalone kserve manifests, to avoid conflicts with 0.6.1 KFServing.
#- kserve_kubeflow.yaml
3 changes: 1 addition & 2 deletions example/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,5 +53,4 @@ resources:
- ../common/user-namespace/base

# KServe
#- ../contrib/kserve/upstream/kserve_kubeflow.yaml
- ../contrib/kserve/upstream/kserve.yaml
- ../contrib/kserve/upstream
23 changes: 23 additions & 0 deletions hack/kind-cluster-1-20.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492
# https://github.com/istio/istio/blob/e02690fbfb8bda564582b27d22d9e8e6e00422a5/prow/config/trustworthy-jwt.yaml#L1-L13
# This configs KinD to spin up a k8s cluster with trustworthy jwt (Service Account Token Volume Projection) feature.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: manifests-1-20
nodes:
- role: control-plane
image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
- role: worker
image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
- role: worker
image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
apiServer:
extraArgs:
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
23 changes: 23 additions & 0 deletions hack/kind-cluster-1-21.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492
# https://github.com/istio/istio/blob/e02690fbfb8bda564582b27d22d9e8e6e00422a5/prow/config/trustworthy-jwt.yaml#L1-L13
# This configs KinD to spin up a k8s cluster with trustworthy jwt (Service Account Token Volume Projection) feature.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: manifests-1-21
nodes:
- role: control-plane
image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
- role: worker
image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
- role: worker
image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
apiServer:
extraArgs:
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
105 changes: 105 additions & 0 deletions hack/setup-kubeflow-light.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash

# Copyright 2021 The Kubeflow Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This shell script is used to setup Katib deployment.
set -euo pipefail

TIMEOUT=600s # 10mins

echo "Creating Kubeflow namespace..."
kubectl create namespace kubeflow

echo "Deploying Cert-Manager."
kustomize build common/cert-manager/cert-manager/base | kubectl apply -f -

echo "Waiting for Cert Manager pods to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n cert-manager --all --for=condition=Ready pod

echo "Deploying Istio."
kustomize build common/istio-1-11/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-11/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-11/istio-install/base | kubectl apply -f -

echo "Waiting for istio-system Pods to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n istio-system --all --for=condition=Ready pod

echo "Deploying Knative."
function install_knative {
kustomize build common/knative/knative-serving/overlays/gateways | kubectl apply -f -
}

while ! install_knative;
do
echo "Retrying to install knative..."
sleep 10
done

kustomize build common/knative/knative-eventing/base | kubectl apply -f -
kustomize build common/istio-1-11/cluster-local-gateway/base | kubectl apply -f -

echo "Waiting for knative-serving Pods to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n knative-serving --all --for=condition=Ready pod

echo "Deploying KFP."
function install_kfp {
kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user | kubectl apply -f - --validate=false
}

while ! install_kfp;
do
echo "Retrying to install kfp..."
sleep 10
done

echo "Waiting for kubeflow/ml-pipelines to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=ml-pipeline --for=condition=Ready pod

echo "Deploying KFServing."
kustomize build apps/kfserving/upstream/overlays/kubeflow | kubectl apply -f -

echo "Waiting for kubeflow/kfserving to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=kfserving --for=condition=Ready pod

echo "Deploying Katib."
kustomize build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -

echo "Waiting for kubeflow/katib to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l katib.kubeflow.org/component=controller --for=condition=Ready pod

echo "Deploying Training Operator."
kustomize build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -

echo "Waiting for kubeflow/training-operator to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l control-plane=kubeflow-training-operator --for=condition=Ready pod

echo "Installing Profiles Controller."
kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -

echo "Waiting for kubeflow/profiles-controller to become ready..."
sleep 5
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l kustomize.component=profiles --for=condition=Ready pod

echo "Creating user resources."
kustomize build common/user-namespace/base | kubectl apply -f -
kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -

58 changes: 58 additions & 0 deletions hack/setup-kubeflow.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/usr/bin/env bash

# Copyright 2021 The Kubeflow Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This shell script is used to setup Katib deployment.
set -euo pipefail

TIMEOUT=600s # 10mins

echo "Creating Kubeflow namespace..."
kubectl create namespace kubeflow

echo "Deploying all Kubeflow components..."
function install_kubeflow {
kustomize build example --load_restrictor none | kubectl apply -f - --validate=false
}

while ! install_kubeflow;
do
echo "Retrying to apply resources"
sleep 10
done

echo "---"
echo "Waiting for all Kubeflow components to become ready."

echo "Waiting for Cert Manager pods to become ready..."
kubectl wait --timeout=${TIMEOUT} -n cert-manager --all --for=condition=Ready pod

echo "Waiting for istio-system Pods to become ready..."
kubectl wait --timeout=${TIMEOUT} -n istio-system --all --for=condition=Ready pod

echo "Waiting for knative-serving Pods to become ready..."
kubectl wait --timeout=${TIMEOUT} -n knative-serving --all --for=condition=Ready pod

echo "Waiting for kubeflow/ml-pipelines to become ready..."
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=ml-pipeline --for=condition=Ready pod

echo "Waiting for kubeflow/kfserving to become ready..."
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=kfserving --for=condition=Ready pod

echo "Waiting for kubeflow/katib to become ready..."
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l katib.kubeflow.org/component=controller --for=condition=Ready pod

echo "Waiting for kubeflow/training-operator to become ready..."
kubectl wait --timeout=${TIMEOUT} -n kubeflow -l control-plane=kubeflow-training-operator --for=condition=Ready pod
72 changes: 0 additions & 72 deletions tests/Makefile

This file was deleted.

9 changes: 0 additions & 9 deletions tests/OWNERS

This file was deleted.

39 changes: 17 additions & 22 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,25 @@
# Kustomize Manifest Tests

## Unittests
## E2E

There are two goals for the unittests:
There are 2 goals for the e2e tests:
1. Verify that core components can be applied and become Ready
2. Virify that core CRDs can be created and succeed

1. Verify that kustomizations are applied correctly (kubeflow/manifests#1014)
1. Verify that various conventions are enforced (kubeflow/manifests#1015)
### Components

### Verifying Kustomizations Are Applied Correctly
The e2e are installing everything by using the [single-install
command](../README.md#install-with-a-single-command). This means that all of
the core and common (Istio, Knativ, Cert Manager) are being installed and
included in the tests.

Examples of kustomizations that we would like to verify are applied and generate the expected output
### Test Suite

* Patches
* Variable substitution
* Composition of resources
The e2e tests are completely independent of the underlying K8s cluster, as well
as the platform of the cluster. These tests should be able to run in real
world clusters, as well as ephemeral ones like KinD.

The general approach to doing this is

1. Check in one more "kustomization.yaml" files corresponding to test cases
1. Run "kustomize build -o ..." and check in the output as the expected test output

* Reviewers can verify changes to the expected output to ensure changes have the desired effect on the expected output
1. Unittests run "kustomize build" and compare output to expected output to ensure kustomize packages are in sync with the expected output
1. Make commands make it easy to regenerate the expected output as part of a change.

```
cd tests
make generate-changed-only
```
The tests are also explicitly bypassing any authentication system of the
installation. The goal of these tests are to ensure that a common use case that
deploys all of Kubeflow's components can succeed. Thus we only want to test
that the core CRDs can be successfully applied and complete.
2 changes: 0 additions & 2 deletions tests/doc.go

This file was deleted.

1 change: 1 addition & 0 deletions tests/e2e/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**/pids.env
Loading