Cherry-picks for v1.5-branch before cutting RC1 (#2138)

* tests: Scripts for e2e tests (#2128)

* remove old test files

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* gitignore: Don't track pyc files

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* flake8: Introduce linting file

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* hack: Introduce scripts for cluster manipulation

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* tests: Add e2e test

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* GH action for running e2e test

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Reduce the installed components and system reqs

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* kserve: Add simple kustomization file

To avoid having to use --load_restrictor none we'll need to wrap the
KServe manifests inside a kustomization.yaml file.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* unittests: Fix unit tests

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* gh: Remove action for e2e tests

We should use prow instead to trigger our e2e tests.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add networkpolicies under /contrib/networkpolicies (#2121)

* Create .gitkeep

* Add files via upload

* Create OWNERS

* Create README.md

* Delete default-deny-not-istio-system.yaml

* Create default-allow-same-namespace.yaml

* Create centraldashboard.yaml

* Create jupyter-web-app.yaml

* Create katib-ui.yaml

* Create kfserving-models-web-app.yaml

* Create ml-pipeline-ui.yaml

* Update ml-pipeline.yaml

* Create volumes-web-app.yaml

* Update kustomization.yaml

* Update OWNERS

* Sync kubeflow pipelines manifests 1.8.0 rc.2 (#2131)

* hack: Update pipelines sync script to change README

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Update kubeflow/pipelines manifests from 1.8.0-rc.2

* Sync kubeflow kubeflow manifests v1.5.0 rc.1 (#2134)

* hack: Sync README for kubeflow/kubeflow sync-script

Extend the sync-script for kubeflow/kubeflow to also update the
components versions in the readme.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Update kubeflow/kubeflow manifests from v1.5.0-rc.1

* Sync kserve/models-web-app manifests (#2135)

* kserve: Rename from upstream to kserve

We will be including both kserve/kserve and kserve/models-web-app into
the manifests, so the names will need to reflect this.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* kserve: Add manifests for the models-web-app

Include the MWA manifests from the v0.7.0 tag.
https://github.com/kserve/models-web-app/tree/v0.7.0

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* kserve: Include both kserve and mwa manifests

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Update kubeflow/kfp-tekton manifests from v1.1.1 (#2141)

* hack: Update tekton script to edit README

The hack script for updating the kfp-tekton manifests should also be
updating the README file as well.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Update kubeflow/kfp-tekton manifests from v1.1.1

* Update manifests for Katib v0.13.0-rc.1 release (#2139)

* Update manifests for Katib v0.13.0-rc.1 release

* Change README

* readme: Remove MPI reference and add ingress distributions link (#2143)

* Closes #1963
* Remove unused MPI reference (PR #2119)

* Update kubeflow/pipelines manifests from 1.8.0 (#2144)

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* hack: Don't error if namespace kubeflow exists (#2140)

The helper setup scripts should not error when the namespaces already
exist.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: a9p <58503488+a9p@users.noreply.github.com>

.flake8

@@ -0,0 +1,4 @@
+[flake8]
+docstring_convention = google
+exclude = assets,__init__.py,__pycache__
+ignore = D100,D103,D104,D107,W503

.github/workflows/manifests_unittests.yaml

@@ -13,9 +13,14 @@ jobs:
     - name: Check out repo
       uses: actions/checkout@v2
 
-    - name: Unit Test
+    - name: Install Kustomize
+      working-directory: ./tests/e2e
       run: |
-        cd tests
-        make test
+        curl -Lo ./kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
+        chmod +x ./kustomize
+        sudo mv kustomize /usr/local/bin
 
+    - name: Unit Test
+      run: |
+        kustomize build example
 

.gitignore

@@ -2,6 +2,8 @@
 .vscode
 .DS_Store
 
+**/*.pyc
+
 # Swap
 [._]*.s[a-v][a-z]
 !*.svg  # comment out if you don't need vector files

README.md

@@ -42,19 +42,19 @@ This repo periodically syncs all official Kubeflow components from their respect
 | Component | Local Manifests Path | Upstream Revision |
 | - | - | - |
 | Training Operator | apps/training-operator/upstream | [v1.4.0-rc.0](https://github.com/kubeflow/tf-operator/tree/v1.4.0-rc.0/manifests) |
-| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/notebook-controller/config) |
-| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/tensorboard-controller/config) |
-| Central Dashboard | apps/centraldashboard/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/centraldashboard/manifests) |
-| Profiles + KFAM | apps/profiles/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/profile-controller/config) |
-| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/admission-webhook/manifests) |
-| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/crud-web-apps/jupyter/manifests) |
-| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/crud-web-apps/tensorboards/manifests) |
-| Volumes Web App | apps/volumes-web-app/upstream | [v1.5.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.0/components/crud-web-apps/volumes/manifests) |
-| Katib | apps/katib/upstream | [v0.13.0-rc.0](https://github.com/kubeflow/katib/tree/v0.12.0-rc.0/manifests/v1beta1) |
+| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/notebook-controller/config) |
+| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/tensorboard-controller/config) |
+| Central Dashboard | apps/centraldashboard/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/centraldashboard/manifests) |
+| Profiles + KFAM | apps/profiles/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/profile-controller/config) |
+| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/admission-webhook/manifests) |
+| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/crud-web-apps/jupyter/manifests) |
+| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/crud-web-apps/tensorboards/manifests) |
+| Volumes Web App | apps/volumes-web-app/upstream | [v1.5.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.5.0-rc.1/components/crud-web-apps/volumes/manifests) |
+| Katib | apps/katib/upstream | [v0.13.0-rc.1](https://github.com/kubeflow/katib/tree/v0.13.0-rc.1/manifests/v1beta1) |
 | KFServing | apps/kfserving/upstream | [v0.6.1](https://github.com/kubeflow/kfserving/releases/tag/v0.6.1) |
 | KServe | contrib/kserve/upstream | [v0.7.0](https://github.com/kserve/kserve/tree/v0.7.0) |
-| Kubeflow Pipelines | apps/pipeline/upstream | [1.8.0-rc.1](https://github.com/kubeflow/pipelines/tree/1.8.0-rc.1/manifests/kustomize) |
-| Kubeflow Tekton Pipelines | apps/kfp-tekton/upstream | [v1.1.0](https://github.com/kubeflow/kfp-tekton/tree/v1.1.0/manifests/kustomize) |
+| Kubeflow Pipelines | apps/pipeline/upstream | [1.8.0](https://github.com/kubeflow/pipelines/tree/1.8.0/manifests/kustomize) |
+| Kubeflow Tekton Pipelines | apps/kfp-tekton/upstream | [v1.1.1](https://github.com/kubeflow/kfp-tekton/tree/v1.1.1/manifests/kustomize) |
 
 The following is also a matrix with versions from common components that are
 used from the different projects of Kubeflow:
@@ -329,14 +329,6 @@ Install the Training Operator official Kubeflow component:
 kustomize build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -
 ```
 
-#### MPI Operator
-
-Install the MPI Operator official Kubeflow component:
-
-```sh
-kustomize build apps/mpi-job/upstream/overlays/kubeflow | kubectl apply -f -
-```
-
 #### User Namespace
 
 Finally, create a new namespace for the the default user (named `kubeflow-user-example-com`).
@@ -376,7 +368,7 @@ After running the command, you can access the Kubeflow Central Dashboard by doin
 
 In order to connect to Kubeflow using NodePort / LoadBalancer / Ingress, you need to setup HTTPS. The reason is that many of our web apps (e.g., Tensorboard Web App, Jupyter Web App, Katib UI) use [Secure Cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies), so accessing Kubeflow with HTTP over a non-localhost domain does not work.
 
-Exposing your Kubeflow cluster with proper HTTPS is a process heavily dependent on your environment. For this reason, please take a look at the available Kubeflow distributions, which are targeted to specific environments, and select the one that fits your needs.
+Exposing your Kubeflow cluster with proper HTTPS is a process heavily dependent on your environment. For this reason, please take a look at the available [Kubeflow distributions](https://www.kubeflow.org/docs/started/installing-kubeflow/#install-a-packaged-kubeflow-distribution), which are targeted to specific environments, and select the one that fits your needs.
 
 ---
 **NOTE**

apps/admission-webhook/upstream/base/kustomization.yaml

@@ -16,7 +16,7 @@ commonLabels:
 images:
 - name: public.ecr.aws/j1r0q0g6/notebooks/admission-webhook
   newName: public.ecr.aws/j1r0q0g6/notebooks/admission-webhook
-  newTag: v1.5.0-rc.0
+  newTag: v1.5.0-rc.1
 namespace: kubeflow
 generatorOptions:
   disableNameSuffixHash: true

apps/centraldashboard/upstream/base/kustomization.yaml

@@ -18,7 +18,7 @@ commonLabels:
 images:
 - name: public.ecr.aws/j1r0q0g6/notebooks/central-dashboard
   newName: public.ecr.aws/j1r0q0g6/notebooks/central-dashboard
-  newTag: v1.5.0-rc.0
+  newTag: v1.5.0-rc.1
 configMapGenerator:
 - envs:
   - params.env

apps/jupyter/jupyter-web-app/upstream/base/cluster-role.yaml

@@ -3,15 +3,6 @@ kind: ClusterRole
 metadata:
   name: cluster-role
 rules:
-- apiGroups:
-  - ""
-  resources:
-  - namespaces
-  verbs:
-  - get
-  - list
-  - create
-  - delete
 - apiGroups:
   - authorization.k8s.io
   resources:

apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml

@@ -17,23 +17,23 @@
 spawnerFormDefaults:
   image:
     # The container Image for the user's Jupyter Notebook
-    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:v1.5.0-rc.0
+    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:v1.5.0-rc.1
     # The list of available standard container Images
     options:
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:v1.5.0-rc.0
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full:v1.5.0-rc.0
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full:v1.5.0-rc.0
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.5.0-rc.0
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full:v1.5.0-rc.0
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:v1.5.0-rc.1
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full:v1.5.0-rc.1
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full:v1.5.0-rc.1
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.5.0-rc.1
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full:v1.5.0-rc.1
   imageGroupOne:
     # The container Image for the user's Group One Server
     # The annotation `notebooks.kubeflow.org/http-rewrite-uri: /`
     # is applied to notebook in this group, configuring
     # the Istio rewrite for containers that host their web UI at `/`
-    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:v1.5.0-rc.0
+    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:v1.5.0-rc.1
     # The list of available standard container Images
     options:
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:v1.5.0-rc.0
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:v1.5.0-rc.1
   imageGroupTwo:
     # The container Image for the user's Group Two Server
     # The annotation `notebooks.kubeflow.org/http-rewrite-uri: /`
@@ -42,10 +42,10 @@ spawnerFormDefaults:
     # The annotation `notebooks.kubeflow.org/http-headers-request-set`
     # is applied to notebook in this group, configuring Istio
     # to add the `X-RStudio-Root-Path` header to requests
-    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:v1.5.0-rc.0
+    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:v1.5.0-rc.1
     # The list of available standard container Images
     options:
-    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:v1.5.0-rc.0
+    - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/rstudio-tidyverse:v1.5.0-rc.1
   # If true, hide registry and/or tag name in the image selection dropdown
   hideRegistry: true
   hideTag: false
@@ -75,71 +75,38 @@ spawnerFormDefaults:
     readOnly: false
   workspaceVolume:
     # Workspace Volume to be attached to user's Notebook
-    # Each Workspace Volume is declared with the following attributes:
-    # Type, Name, Size, MountPath and Access Mode
+    # If you don't want a workspace volume then delete the 'value' key
     value:
-      type:
-        # The Type of the Workspace Volume
-        # Supported values: 'New', 'Existing'
-        value: New
-      name:
-        # The Name of the Workspace Volume
-        # Note that this is a templated value. Special values:
-        # {notebook-name}: Replaced with the name of the Notebook. The frontend
-        #                  will replace this value as the user types the name
-        value: 'workspace-{notebook-name}'
-      size:
-        # The Size of the Workspace Volume (in Gi)
-        value: '5Gi'
-      mountPath:
-        # The Path that the Workspace Volume will be mounted
-        value: /home/jovyan
-      accessModes:
-        # The Access Mode of the Workspace Volume
-        # Supported values: 'ReadWriteOnce', 'ReadWriteMany', 'ReadOnlyMany'
-        value: ReadWriteOnce
-      class:
-        # The StrageClass the PVC will use if type is New. Special values are:
-        # {none}: default StorageClass
-        # {empty}: empty string ""
-        value: '{none}'
+      mount: /home/jovyan
+      newPvc:
+        metadata:
+          name: '{notebook-name}-workspace'
+        spec:
+          resources:
+            requests:
+              storage: 10Gi
+          accessModes:
+          - ReadWriteOnce
     readOnly: false
   dataVolumes:
     # List of additional Data Volumes to be attached to the user's Notebook
     value: []
-    # Each Data Volume is declared with the following attributes:
-    # Type, Name, Size, MountPath and Access Mode
-    #
     # For example, a list with 2 Data Volumes:
     # value:
-    #   - value:
-    #       type:
-    #         value: New
-    #       name:
-    #         value: '{notebook-name}-vol-1'
-    #       size:
-    #         value: '10Gi'
-    #       class:
-    #         value: standard
-    #       mountPath:
-    #         value: /home/jovyan/vol-1
-    #       accessModes:
-    #         value: ReadWriteOnce
-    #       class:
-    #         value: {none}
-    #   - value:
-    #       type:
-    #         value: New
-    #       name:
-    #         value: '{notebook-name}-vol-2'
-    #       size:
-    #         value: '10Gi'
-    #       mountPath:
-    #         value: /home/jovyan/vol-2
-    #       accessModes:
-    #         value: ReadWriteMany
-    #       class:
-    #         value: {none}
+    #   - mount: /home/jovyan/datavol-1
+    #     newPvc:
+    #       metadata:
+    #         name: '{notebook-name}-datavol-1'
+    #       spec:
+    #         resources:
+    #           requests:
+    #             storage: 5Gi
+    #         accessModes:
+    #           - ReadWriteOnce
+    #   - mount: /home/jovyan/datavol-1
+    #     existingSource:
+    #       persistentVolumeClaim:
+    #         claimName: test-pvc
     readOnly: false
   gpus:
     # Number of GPUs to be assigned to the Notebook Container

apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml

@@ -23,7 +23,7 @@ commonLabels:
 images:
 - name: public.ecr.aws/j1r0q0g6/notebooks/jupyter-web-app
   newName: public.ecr.aws/j1r0q0g6/notebooks/jupyter-web-app
-  newTag: v1.5.0-rc.0
+  newTag: v1.5.0-rc.1
 # We need the name to be unique without the suffix because the original name is what
 # gets used with patches
 configMapGenerator:

apps/jupyter/jupyter-web-app/upstream/base/role-binding.yaml

@@ -1,4 +1,4 @@
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
   name: jupyter-notebook-role-binding

apps/jupyter/jupyter-web-app/upstream/base/role.yaml

@@ -1,35 +1,48 @@
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: jupyter-notebook-role
 rules:
 - apiGroups:
-  - ""
+  - authorization.k8s.io
   resources:
-  - pods
-  - pods/log
-  - secrets
-  - services
+  - subjectaccessreviews
   verbs:
-  - '*'
+  - create
+- apiGroups:
+  - kubeflow.org
+  resources:
+  - notebooks
+  - notebooks/finalizers
+  - poddefaults
+  verbs:
+  - get
+  - list
+  - create
+  - delete
+  - patch
+  - update
 - apiGroups:
   - ""
-  - apps
-  - extensions
   resources:
-  - deployments
-  - replicasets
+  - persistentvolumeclaims
   verbs:
-  - '*'
+  - create
+  - delete
+  - get
+  - list
 - apiGroups:
-  - kubeflow.org
+  - ""
   resources:
-  - '*'
+  - events
+  - nodes
   verbs:
-  - '*'
+  - list
 - apiGroups:
-  - batch
+  - storage.k8s.io
   resources:
-  - jobs
+  - storageclasses
   verbs:
-  - '*'
+  - get
+  - list
+  - watch

apps/jupyter/notebook-controller/upstream/base/kustomization.yaml

@@ -5,4 +5,4 @@ resources:
 images:
 - name: public.ecr.aws/j1r0q0g6/notebooks/notebook-controller
   newName: public.ecr.aws/j1r0q0g6/notebooks/notebook-controller
-  newTag: v1.5.0-rc.0
+  newTag: v1.5.0-rc.1