Kubeflow 1.4->1.6 Rebase

OWNERS

@@ -1,5 +1,6 @@
 # We want to rely on fine grained OWNERs files.
 # Approvers listed here is just a backup in case someone is out.
 approvers:
-  - Bobgy
-  - pdmack
+- pdmack
+- james-jwu
+- zijianjoy

README.md

@@ -18,10 +18,9 @@ The Kubeflow community is organized into working groups (WGs) with associated re
 * [Training](https://github.com/kubeflow/community/tree/master/wg-training)
 
 ## Quick Links
-* [Prow test dashboard](https://k8s-testgrid.appspot.com/sig-big-data)
-* [Prow jobs dashboard](https://prow.k8s.io/?repo=kubeflow%2Fkubeflow)
+* [Prow jobs dashboard](http://prow.kubeflow-testing.com)
 * [PR Dashboard](https://k8s-gubernator.appspot.com/pr)
-* [Argo UI for E2E tests](http://testing-argo.kubeflow.org)
+* [Argo UI for E2E tests](https://argo.kubeflow-testing.com)
 
 ## Get Involved
 Please refer to the [Community](https://www.kubeflow.org/docs/about/community/) page.

ROADMAP.md

@@ -1,6 +1,102 @@
 # Kubeflow Roadmap
 
-## Kubeflow 1.4 Release, Due: September 2021
+## Kubeflow 1.6 Release, Due: July 2022
+* Kubeflow 1.6 [milestones and timelines](https://github.com/kubeflow/community/pull/558)
+
+#### Themes
+* Kubernetes 1.22 support
+
+### Major Features from each Working Group (note: Individual WG versions are independent of Kubeflow's)
+
+#### Kubeflow Pipelines
+* V2 Preview: Support running pipeline in Argo-agnostic approach
+* New DAG visualization based on Pipeline Template and MLMD
+* SDK to change from `kfp.v2` to `kfp`
+* KFP SDK CLI [design](https://docs.google.com/document/d/1Z79ebKaf3hjyadSRsme1Qo7F-Tqu4qM1cyKuLeu2PgA/edit#heading=h.x9snb54sjlu9)
+
+#### Katib
+* Support for Population based training
+* Support JSON format for Metrics Collector
+* UI Enhancement to add a separate page for Trials and show Trial logs
+
+#### Training Operator
+* PyTorch elastic training enhancements
+* Elastic scheduling with Volcano support
+* MPI Operator v2
+
+#### Notebooks, Central Dashboard, Web Apps, Manifest
+* Sorting and filtering support for the web apps
+* Expose Notebook idleness information in the JWA
+
+#### KServe
+* Knative 1.0 support and certified for KServe Serverless installation
+* KServe 0.9 SDK with ModelServer API name changes and improvement for CloudEvent support
+* New ServingRuntime and ClusterServingRuntime CRDs
+* New ModelSpec in InferenceServices
+* ModelMesh updates include support for multi-namespace reconciliation
+
+#### Detailed features, bug fixes and enhancements will be identified in the Working Group Roadmaps
+
+* [Training Operators](https://github.com/kubeflow/common/blob/master/ROADMAP.md)
+* [KFServing](https://github.com/kubeflow/kfserving/blob/master/ROADMAP.md)
+* [Katib](https://github.com/kubeflow/katib/blob/master/ROADMAP.md)
+* [Kubeflow Pipelines](https://github.com/kubeflow/pipelines/blob/master/ROADMAP.md)
+* [Notebooks, Central Dashboard](https://docs.google.com/document/d/1YtSWRhdhyOgd6ZQcWLM38TGDy2H_EhXjr8U5lUi37_I/edit)
+
+
+## Kubeflow 1.5 Release, Delivered: March 2022
+* Kubeflow 1.5 [milestones and timeline](https://github.com/kubeflow/community/pull/538)
+
+### Themes
+* Switching to [Emissary executor](https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#emissary-executor) enables Kubeflow Pipelines deployment on Kubernetes >= v1.20, which runs on containerd runtime instead of Docker runtime
+* Improve model accuracy and reduce overfitting, especially with hyper parameter tuning
+* Simplify operations and optimize utilization (including spot instance use cases for distributed training) 
+* More consistent user experience - UI appearance, features and naming
+* Improved documentation, tutorials and examples
+* Stretch - Support for K8s 1.22 and associated dependencies (cert mgr, istio)
+
+### Major Features from each Working Group (note: Individual WG versions are independent of Kubeflow's)
+
+#### Kubeflow Pipelines, v1.8
+* Switching to [Emissary executor](https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#emissary-executor) enables Kubeflow Pipelines deployment on Kubernetes >= v1.20, which runs on containerd runtime instead of Docker runtime.
+
+#### Katib, v0.13 
+* Katib controller leader election for HA operations and faster recovery
+* Validation for Early Stopping algorithm settings helps users to reduce model overfitting
+* Improve SDK, AWS CI, parameter settings across frameworks (goptuna, optuna, hyperopt) 
+* Update namespace label for Metrics Collector injection (note - breaking change)
+
+#### Training Operator, v1.3
+* Elastic Training for PyTorch, restarts pod rather than whole job, supports spot instances
+* MPI addition to Unified training operator 
+* Python SDK supported on PyPI 
+
+#### Notebooks, Central Dashboard, Web Apps, Manifest, v1.5 
+* Exposing notebook idleness parameters, reduces infra used on idle notebooks
+* UI consistency between web apps along with frontend pagination, and remove cards and user responsive tables
+* Support for dark mode
+* More options to define PVC specification
+
+#### KServe, v0.7
+* KFServing is rebranded to [KServe](https://github.com/kserve)
+* [ModelMesh](https://github.com/kserve/modelmesh-serving) joins KServe 
+* Triton model serving runtime defaults to v21.09
+* (Alpha feature) Raw kubernetes deployment support, Istio/Knative dependency is now optional
+
+#### Detailed features, bug fixes and enhancements are identified in the Working Group Roadmaps
+
+* [Training Operators](https://github.com/kubeflow/common/blob/master/ROADMAP.md)
+* [KFServing](https://github.com/kubeflow/kfserving/blob/master/ROADMAP.md)
+* [Katib](https://github.com/kubeflow/katib/blob/master/ROADMAP.md)
+* [Kubeflow Pipelines](https://github.com/kubeflow/pipelines/blob/master/ROADMAP.md)
+* [Notebooks](https://github.com/kubeflow/kubeflow/issues/5978)
+
+
+## Kubeflow 1.4.1 Release, Delivered: December 2021
+* The need for a patch release was triggered by [#2082](https://github.com/kubeflow/manifests/issues/2082)
+* 1.4.1 Tracking issue: [2084](https://github.com/kubeflow/manifests/issues/2082)
+
+## Kubeflow 1.4 Release, Delivered: October 2021
 
 ### Themes and Major Features
 

components/Makefile

@@ -0,0 +1,46 @@
+REGISTRY ?= kubeflownotebookswg
+TAG ?= $(shell git describe --tags --always --dirty)
+
+build-all:
+	@echo "\nBuilding KFAM image...\n"
+	make docker-build -C access-management 
+
+	@echo "\nBuilding Admission-webhook image...\n"
+	make docker-build -C admission-webhook
+
+	@echo "\nBuilding Centraldashboard image...\n"
+	make docker-build -C centraldashboard
+
+	@echo "\nBuilding Jupyter-Web-App image...\n"
+	make docker-build -C crud-web-apps/jupyter
+
+	@echo "\nBuilding Tensorboard-Web-App image...\n"
+	make docker-build -C crud-web-apps/tensorboards
+
+	@echo "\nBuilding Volumes-Web-App image...\n"
+	make docker-build -C crud-web-apps/volumes
+
+	@echo "\nBuilding Notebook-controller image...\n"
+	make docker-build -C notebook-controller
+
+	@echo "\nBuilding Profile-controller image...\n"
+	make docker-build -C profile-controller
+
+	@echo "\nBuilding tensorboard-controller image...\n"
+	make docker-build -C tensorboard-controller
+
+	@echo "\nBuilding notebook-server images...\n"
+	make docker-build-all -C example-notebook-servers
+
+	@echo "\nAll images of prv-kubeflow-kubeflow repo have been built successfully!\n"
+
+push-all:	
+	for img in kfam admission-webhook centraldashboard jupyter-web-app tensorboards-web-app volumes-web-app notebook-controller profile-controller \
+	tensorboard-controller ; do \
+		docker tag $$img:$(TAG) ${REGISTRY}/$$img:${TAG} ; \
+		docker push ${REGISTRY}/$$img:${TAG} ; \
+	done
+
+	make docker-push-all -C example-notebook-servers TAG=${TAG} REGISTRY=${REGISTRY}
+
+	@echo "\nAll images have been succesfully retaged and pushed with registry: ${REGISTRY} and tag: ${TAG}...\n"

components/access-management/Makefile

@@ -1,30 +1,24 @@
 REGISTRY_PROJECT ?= kubeflow-dev
-GOLANG_VERSION ?= 1.12
-GCLOUD_PROJECT ?= kubeflow-images-public
-IMG ?= gcr.io/$(GCLOUD_PROJECT)/kfam
 PROJECT ?= kubeflow-dev
-TAG ?= $(eval TAG := $(shell date +v%Y%m%d)-$(shell git describe --tags --always --dirty)-$(shell git diff | shasum -a256 | cut -c -6))$(TAG)
+GO_SWAGGER_URL ?= https://github.com/go-swagger/go-swagger/releases/download/v0.21.0/swagger_linux_amd64
 
-CHANGED_FILES := $(shell git diff-files --relative=components/centraldashboard)
+GIT_VERSION := $(shell git describe --tags --always --dirty)
 
-ifeq ($(strip $(CHANGED_FILES)),)
-# Changed files is empty; not dirty
-# Don't include --dirty because it could be dirty if files outside the ones we care
-# about changed.
-GIT_VERSION := $(shell git describe --tags --long)
-else
-GIT_VERSION := $(shell git describe --tags --long)-dirty-$(shell git diff | shasum -a256 | cut -c -6)
-endif
+IMG ?= kfam
+TAG ?= $(GIT_VERSION)
 
-build:
-	docker build -t $(IMG):$(TAG) .
+generate-go-client: bin/swagger
+	bin/swagger generate client -f api/swagger.yaml -t api/go_client
+
+bin/swagger:
+	mkdir -p bin
+	wget -O bin/swagger '$(GO_SWAGGER_URL)'
+	chmod +x bin/swagger
 
-build-gcb:
-	gcloud --project=$(PROJECT) \
-		builds submit \
-		--machine-type=n1-highcpu-32 \
-		--substitutions=_GIT_VERSION=$(GIT_VERSION),_REGISTRY=$(REGISTRY_PROJECT) \
-		--config=cloudbuild.yaml .
+docker-build:
+	docker build -t $(IMG):$(TAG) .
 
-push: build
+docker-push: 
 	docker push $(IMG):$(TAG)
+
+image: docker-build docker-push 

components/access-management/README.md

@@ -8,7 +8,7 @@ The goal is to support multi-tenancy kubeflow cluster / services.
 ### Profile
 - Profile contains owner which refers to a k8s user.
 - Profile will create a namespace with same name and make profile owner the namespace owner.
-- Profile will create necessary k8s resources to enable owner access to kubeflow servuces under multi-tenancy mode.
+- Profile will create necessary k8s resources to enable owner access to kubeflow services under multi-tenancy mode.
 - After the namespace is created, the owner can grant access to additional users or groups using the API or by creating RBAC roles & bindings directly.
 - Delete profile will delete namespace and other k8s resources owned by this profile.
 

components/admission-webhook/Makefile

@@ -1,11 +1,17 @@
 # Image URL to use all building/pushing image targets
-IMG ?= gcr.io/kubeflow-images-public/admission-webhook
-TAG ?= $(eval TAG := $(shell date +v%Y%m%d)-$(shell git describe --tags --always --dirty)-$(shell git diff | shasum -a256 | cut -c -6))$(TAG)
-GOLANG_VERSION ?= 1.12
+IMG ?= admission-webhook
+TAG ?= $(shell git describe --tags --always --dirty)
 
 build:
 	go build -gcflags 'all=-N -l' -o bin/webhook .
 
-build-gcr:
-	docker build -t $(IMG):$(TAG) .
-	@echo Built $(IMG):$(TAG)
+docker-build:
+	docker build -t ${IMG}:${TAG} -f Dockerfile .
+
+docker-push: 
+	docker push ${IMG}:${TAG}
+
+image: docker-build docker-push
+
+docker-push:
+	docker push ${IMG}:${TAG}

components/admission-webhook/README.md

@@ -1,8 +1,8 @@
 ## Goal
 We need a way to inject common data (env vars, volumes) to pods (e.g. notebooks).
 See [issue](https://github.com/kubeflow/kubeflow/issues/2641).
-K8s has [PodPreset](https://kubernetes.io/docs/concepts/workloads/pods/podpreset/) resource with similar use-case, however it is in alpha. 
-K8s [admission-controller](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhookConfiguration) and CRD can be used to implement PodPreset as done in [here](https://github.com/jpeeler/podpreset-crd).
+K8s has [PodPreset](https://v1-19.docs.kubernetes.io/docs/concepts/workloads/pods/podpreset/) resource with similar use-case, however it is in alpha. 
+K8s [admission-controller](https://godoc.org/k8s.io/api/admissionregistration/v1#MutatingWebhookConfiguration) and CRD can be used to implement PodPreset as done in [here](https://github.com/jpeeler/podpreset-crd).
 We borrowed this PodPreset implementation, customize it for Kubeflow and rename it to PodDefault to avoid confusion.  
 The code is not directly used as Kubeflow's use case for PodDefault controller is slightly different. 
 In fact, PodDefault in Kubeflow is defined as CRD without the  custom controller (as opposed to [here](https://github.com/jpeeler/podpreset-crd)).
@@ -13,7 +13,7 @@ Here is the workflow on how this is used in Kubeflow:
 1. Users create  PodDefault manifests which describe additional runtime requirements (i.e., volume, volumeMounts, environment variables) to be injected  into a Pod at creation time.
 PodDefaults use label selectors to specify the Pods to which a given PodDefault applies.
 PodDefaults are namespace scope, i.e., they can be applied/viewed in the namespace.  
-As an example, the following manifest declares a PodPrest in `kubeflow` namespace to add the secret ```gcp-secret``` in to pods in the given namespace. 
+As an example, the following manifest declares a `PodDefault` in the `kubeflow` namespace to add the secret ```gcp-secret``` in to pods in the given namespace. 
 
 ```
 apiVersion: "kubeflow.org/v1alpha1"
@@ -48,11 +48,11 @@ For the above PodDefault, when a pod creation request comes which has the label
 to the pod as described in the PodDefault spec.
 
 ## Webhook Configuration
-Define a [MutatingWebhookConfiguration](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhookConfiguration),
+Define a [MutatingWebhookConfiguration](https://godoc.org/k8s.io/api/admissionregistration/v1#MutatingWebhookConfiguration),
 for example:
 
 ```
-apiVersion: admissionregistration.k8s.io/v1beta1
+apiVersion: admissionregistration.k8s.io/v1
 kind: MutatingWebhookConfiguration
 metadata:
   name: gcp-cred-webhook
@@ -75,17 +75,24 @@ This specifies
 1. When there is a pod being created (see `rules`),
 1. call the webhook service `gcp-cred-webhook.default` at path `/add-cred` (see `clientConfig`)
 
+### admissionregistration.k8s.io/v1 default failurePolicy
+In adopting `admissionregistration.k8s.io/v1` for the `MutatingWebhookConfiguration` we accept the default value
+for `failurePolicy` to be `Fail` per [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy). Upon testing this default feature it was discovered if the `AdmissionWebhook`'s `mutating-webhook-configuration` failed to mutate a pod then the pod would fail to start, its associated `Deployment` or `StatefulSet` would continually attempt to create the pod until the process that created the pod was terminated. Furthermore, any pod that is targeted by the `AdmissionWebhook` for mutation was effected and will fail to start if there are configuration collisions or some other issue, already running pods are not effected by this policy. Again, only pods created after the policy change may be effected.
+
+Engineers should be mindful of this setting as it was also noted these failures can persist beyond the `timeoutSeconds` parameter that
+is by default set to `10` seconds in the `v1` API. For example, a `StatefulSet`s attempted to create notebook pods
+despite its API request getting rejected by the `mutating-webhook-configuration`. Please refer to kubernetes documentation to read up on
+the `failurePolicy: Ignore` parameter as this was the default value in `v1beta1`.
 
 ### Webhook implementation
 The webhook should be a server that can handle request coming from the configured path (`/add-cred` in the above).
-The request and response types are both [AdmissionReview](https://godoc.org/k8s.io/api/admission/v1beta1#AdmissionReview)
+The request and response types are both [AdmissionReview](https://godoc.org/k8s.io/api/admission/v1#AdmissionReview)
 
 ## Reference
-1. [K8S PodPreset](https://kubernetes.io/docs/concepts/workloads/pods/podpreset)
+1. [K8S PodPreset](https://v1-19.docs.kubernetes.io/docs/concepts/workloads/pods/podpreset/)
 1. https://github.com/jpeeler/podpreset-crd
 1. https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
 1. https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/webhook
 1. https://github.com/morvencao/kube-mutating-webhook-tutorial
 1. How to self sign: [link](https://github.com/kubernetes/kubectl/issues/86)
 1. What to put for caBundle: [issue](https://github.com/kubernetes/kubernetes/issues/61171)
-

components/admission-webhook/config.go

@@ -17,11 +17,13 @@ limitations under the License.
 package main
 
 import (
+	"context"
 	"crypto/tls"
 
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	"k8s.io/klog"
+	"sigs.k8s.io/controller-runtime/pkg/certwatcher"
 	"sigs.k8s.io/controller-runtime/pkg/client/config"
 )
 
@@ -39,12 +41,19 @@ func getCrdClient() client.Client {
 }
 
 func configTLS(config Config) *tls.Config {
-	sCert, err := tls.LoadX509KeyPair(config.CertFile, config.KeyFile)
+	certWatcher, err := certwatcher.New(config.CertFile, config.KeyFile)
 	if err != nil {
 		klog.Fatalf("config=%#v Error: %v", config, err)
 	}
+
+	go func() {
+		if err := certWatcher.Start(context.TODO()); err != nil {
+			klog.Fatalf("config=%#v Error: %v", config, err)
+		}
+	}()
+
 	return &tls.Config{
-		Certificates: []tls.Certificate{sCert},
+		GetCertificate: certWatcher.GetCertificate,
 		// TODO: uses mutual tls after we agree on what cert the apiserver should use.
 		//ClientAuth: tls.RequireAndVerifyClientCert,
 	}