Add Notebook Culling Feature Doc (#490)

**Which issue is resolved by this Pull Request:** Resolves # #439 **Description of your changes:** -Added documentation in notebook component guide for culling feature
awslabs · Nov 22, 2022 · 3e605fb · 3e605fb
1 parent f69753d
commit 3e605fb
Show file tree

Hide file tree

Showing 10 changed files with 76 additions and 0 deletions.
diff --git a/website/content/en/docs/deployment/cognito-rds-s3/guide-terraform.md b/website/content/en/docs/deployment/cognito-rds-s3/guide-terraform.md
@@ -102,6 +102,9 @@ pwd
     EOF
     ```
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 ### View all Configurations
 
 View [all possible configuration options of the terraform stack](https://github.com/awslabs/kubeflow-manifests/blob/main/deployments/cognito-rds-s3/terraform/variables.tf) in the `variables.tf` file.

diff --git a/website/content/en/docs/deployment/cognito-rds-s3/guide.md b/website/content/en/docs/deployment/cognito-rds-s3/guide.md
@@ -26,6 +26,10 @@ Refer to the [general prerequisites guide]({{< ref "/docs/deployment/prerequisit
     1. Create TLS certificates for the domain
     1. Create a Cognito Userpool
     1. Configure Ingress
+
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 2. Deploy Kubeflow.
     1. Install Kubeflow using the following command:
 {{< tabpane persistLang=false >}}

diff --git a/website/content/en/docs/deployment/cognito/guide-terraform.md b/website/content/en/docs/deployment/cognito/guide-terraform.md
@@ -80,6 +80,9 @@ pwd
     EOF
     ```
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 ### View all configurations
 
 View [all possible configuration options of the terraform stack](https://github.com/awslabs/kubeflow-manifests/blob/main/deployments/cognito/terraform/variables.tf) in the `variables.tf` file.

diff --git a/website/content/en/docs/deployment/cognito/manifest/guide-automated.md b/website/content/en/docs/deployment/cognito/manifest/guide-automated.md
@@ -102,6 +102,9 @@ Each section is detailed in [Cognito Manual Deployment Guide]({{< ref "/docs/dep
             us-east-1-certARN: arn:aws:acm:us-east-1:123456789012:certificate/373cc726-f525-4bc7-b7bf-d1d7b641c238
     ```
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide. 
+
 ## 2.0 Install Kubeflow
 
 Install Kubeflow using the following command:

diff --git a/website/content/en/docs/deployment/cognito/manifest/guide.md b/website/content/en/docs/deployment/cognito/manifest/guide.md
@@ -157,6 +157,9 @@ yq e '.LOGOUT_URL = env(CognitoLogoutURL)' -i charts/common/aws-authservice/valu
 
 1. Follow the [Configure Load Balancer Controller]({{< ref "/docs/add-ons/load-balancer/guide.md#configure-load-balancer-controller" >}}) section of the load balancer guide to setup the resources required by the load balancer controller.
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 ## 4.0 Build the manifests and deploy Kubeflow
 
 {{< tabpane persistLang=false >}}

diff --git a/website/content/en/docs/deployment/configure-notebook-culling.md b/website/content/en/docs/deployment/configure-notebook-culling.md
@@ -0,0 +1,47 @@
++++
+title = "Configure Culling for Notebooks"
+description = "Automatically stop your notebooks based on idleness"
+weight = 80
++++
+
+ The culling feature allows you to stop a Notebook Server based on its **Last Activity**. The Notebook Controller updates the respective `notebooks.kubeflow.org/last-activity` annotation of each Notebook resource according to the execution state of the kernels. When this feature is enabled, the notebook instances will be "culled" (scaled to zero) if none of the kernels are performing computations for a specified period of time (`CULL_IDLE_TIME`). More information about this feature can be found in the [Jupyter notebook idleness proposal](https://github.com/kubeflow/kubeflow/blob/master/components/proposals/20220121-jupyter-notebook-idleness.md).
+
+1. Export the following values values to configure the [culling policy parameters](https://github.com/kubeflow/kubeflow/blob/master/components/proposals/20220121-jupyter-notebook-idleness.md#api-changes):
+    ```bash
+    # whether to enable culling feature (true/false). ENABLE_CULLING must be set to “true” for this feature to take work
+    export ENABLE_CULLING="true"
+    # specified idleness time (minutes) that notebook instance to be culled since last activity
+    export CULL_IDLE_TIMEOUT="30"
+    # controller will update each notebook's LAST_ACTIVITY_ANNOTATION every IDLENESS_CHECK_PERIOD (minutes)
+    export IDLENESS_CHECK_PERIOD="5"
+    ```
+
+1. The following commands will inject those values in a configuration file for setting up Notebook culling:
+    Select the package manager of your choice.
+    - For Kustomize and Helm:
+    {{< tabpane persistLang=false >}}
+    {{< tab header="Kustomize" lang="sh" >}}
+printf '
+enableCulling='$ENABLE_CULLING'
+cullIdleTime='$CULL_IDLE_TIMEOUT'
+idlenessCheckPeriod='$IDLENESS_CHECK_PERIOD'
+' > awsconfigs/apps/notebook-controller/params.env
+    {{< /tab >}}
+    {{< tab header="Helm" lang="sh" >}}
+yq e '.cullingPolicy.enableCulling = env(ENABLE_CULLING)' -i charts/apps/notebook-controller/values.yaml
+yq e '.cullingPolicy.cullIdleTime = env(CULL_IDLE_TIMEOUT)' -i charts/apps/notebook-controller/values.yaml
+yq e '.cullingPolicy.idlenessCheckPeriod = env(IDLENESS_CHECK_PERIOD)' -i charts/apps/notebook-controller/values.yaml
+    {{< /tab >}}
+    {{< /tabpane >}}
+
+    - For Terraform, append the notebook culling parameters in the `sample.auto.tfvars` file with chosen deployment option: [Vanilla]({{< ref "/docs/deployment/vanilla/guide-terraform.md#" >}}), [Cognito]({{< ref "/docs/deployment/cognito/guide-terraform.md#" >}}), [RDS-S3]({{< ref "/docs/deployment/rds-s3/guide-terraform.md#" >}}), and [Cognito-RDS-S3]({{< ref "/docs/deployment/cognito-rds-s3/guide-terraform.md#" >}}).
+
+        ```sh
+        cat <<EOT >> sample.auto.tfvars
+        notebook_enable_culling="${ENABLE_CULLING}" 
+        notebook_cull_idle_time="${CULL_IDLE_TIMEOUT}"
+        notebook_idleness_check_period="${IDLENESS_CHECK_PERIOD}"
+        EOT
+        ```
+
+1. Continue deploying Kubeflow based on your [Deployment Option]({{< ref "/docs/deployment/_index.md#" >}}).
diff --git a/website/content/en/docs/deployment/rds-s3/guide-terraform.md b/website/content/en/docs/deployment/rds-s3/guide-terraform.md
@@ -77,6 +77,9 @@ pwd
     EOF
     ```
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 ### All Configurations
 
 A full list of inputs for the terraform stack can be found [here](https://github.com/awslabs/kubeflow-manifests/blob/main/deployments/rds-s3/terraform/variables.tf).

diff --git a/website/content/en/docs/deployment/rds-s3/guide.md b/website/content/en/docs/deployment/rds-s3/guide.md
@@ -244,6 +244,9 @@ yq e '.s3.minioServiceRegion = env(CLUSTER_REGION)' -i charts/apps/kubeflow-pipe
             {{< /tab >}}
             {{< /tabpane >}}
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide. 
+
 
 ## 3.0 Build Manifests and install Kubeflow
 

diff --git a/website/content/en/docs/deployment/vanilla/guide-terraform.md b/website/content/en/docs/deployment/vanilla/guide-terraform.md
@@ -54,6 +54,9 @@ pwd
     EOF
     ```
 
+### (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
 ### All Configurations
 
 A full list of inputs for the terraform stack can be found [here](https://github.com/awslabs/kubeflow-manifests/blob/main/deployments/vanilla/terraform/variables.tf).

diff --git a/website/content/en/docs/deployment/vanilla/guide.md b/website/content/en/docs/deployment/vanilla/guide.md
@@ -14,6 +14,10 @@ Be sure that you have satisfied the installation prerequisites before working th
 - [Set up your deployment environment]({{< ref "prerequisites.md" >}})
 - [Create an EKS Cluster]({{< ref "create-eks-cluster.md" >}})
 
+## (Optional) Configure Culling for Notebooks
+Enable culling for notebooks by following the [instructions]({{< ref "/docs/deployment/configure-notebook-culling.md#" >}}) in configure culling for notebooks guide.
+
+
 ## Build Manifests and install Kubeflow
 
 > ⚠️ Warning: We use a default email (`user@example.com`) and password (`12341234`) for our guides. For any production Kubeflow deployment, you should change the default password by following the steps in [Change default user password]({{< ref "../connect-kubeflow-dashboard#change-the-default-user-password-kustomize" >}}).