kubeflow v1.1/1.2 upstream changes to jupyter-apis component

[saffaalvi]

Summary of upstream changes that we may have to integrate into the jupyter-apis component when upgrading to v1.1/1.2.

[saffaalvi]

Kubeflow v1.1

Overall, it has been difficult to find a comprehensive list of all the new features Kubeflow v1.1 has introduced. The changes mentioned below are what I've noticed from v1.1 branch commits in the jupyter-web-app component.

This blog post talks more in detail about the changes in v1.1. I think the jupyter-web-app component didn't have as many changes in this upgrade.

Notebooks Table

• Added a GPU column to table
• Added a distinct function to process GPU details (which we already implemented)
• PR: Added gpu details to the notebook server table kubeflow/kubeflow#5074

Volume (Resource Form)

• Added an volume dropdown to show all existing PVCs under volumes for a new notebook server (which we have already implemented): show all existing PVCs as dropdown under volumes for a new notebook server kubeflow/kubeflow#5086
• Fix UI handling of default StorageClass: JWA: Fix UI handling of default StorageClass kubeflow/kubeflow#4907

[saffaalvi]

Kubeflow v1.2

After Kubeflow v1.1 was released, the major change was a new directory for web apps that shared common code (includes Jupyter, Tensorboard, and Volumes UI). There are still commits being made to both the old jupyter-web-app component as well as the new crud-web-apps/jupyter component. I described all the changes made to both below but tried to keep them separate to avoid confusion.

There is more information for what v1.2 will have in their release description https://github.com/kubeflow/kfctl/releases/tag/v1.2-rc.0 and this roadmap issue kubeflow/kubeflow#5224.

Some features mentioned below were found from the following PRs:
Not merged: kubeflow/kubeflow#5280
Merged: kubeflow/kubeflow#5332

Old jupyter-web-app Component

• Trim whitespace from custom images: Fix #5111: Trim whitespace from custom images kubeflow/kubeflow#5112
• Upgrade jupyter-web-app to Angular 8: Upgrade jupyter-web-app to Angular 8 kubeflow/kubeflow#5249 (we're about 2 versions ahead of them)
• Pod resources limit: issue was created, and a PR was merged but there was still a conversation about whether this change was the best.

  • The jupyter-web-app rewrite PR jupyter-web-app rewrite kubeflow/kubeflow#5280 has put this feature behind config flags which are disabled by default. There's also a warning displayed if they're enabled. Not sure if this feature has been included in this release yet.
    

  • There was then a separate PR Add support for Tolerations and Affinity in Notebooks kubeflow/kubeflow#5237 to add support for toleration and affinity configs in the Notebook Spawner UI. Adds options to choose an affinity config and toleration config group under data volumes when creating a new notebook server.
    

New Directory for Common Web Apps (crud-web-apps)

• Created a new directory in components for web apps PR: Create a new directory in components for web apps kubeflow/kubeflow#5184 (Jupyter, Tensorboard, and Volumes UIs), called crud-web-apps
• Common code between the different python backends PR: Common code between the different python backends kubeflow/kubeflow#5164
• Create an Angular Library with common frontend code: Create an Angular Library with common frontend code kubeflow/kubeflow#5252
• Refactor the JWA backend to utilize common code: Refactor the JWA backend to utilize common code kubeflow/kubeflow#5316
  • Added the support for tolerations and affinity here as well
• Initialize the Jupyter web app frontend in crud-web-apps: Initialize the Jupyter web app frontend in crud-web-apps kubeflow/kubeflow#5332
  • Start/Stop Notebooks
  • Advanced options to allow user to specify image pull policy of the Notebook's image
  • Don't allow selecting GPUs that aren't installed
• It seems a Persistent Volume Management UI PR was created and is still open: Persistent Volume Management UI kubeflow/kubeflow#4758. In most of the PRs above, they mention a Volumes web app, but I think it is still a work in progress as there is no merged PR specifically for this new feature (likely won't be in this release then).
• A major PR for the jupyter-web-app rewrite with a lot of new changes was created but not merged jupyter-web-app rewrite kubeflow/kubeflow#5280. It seems this PR was broken down into smaller fixes to the new web app (aka the PRs in this list).

[saffaalvi]

MiniKF

When running MiniKF, these are some changes I've seen that I'm not sure have been/will be implemented in v1.1 or v1.2:

Notebooks Table

• Can now interact more with notebook servers: start, stop, or delete (options menu instead of trash icon)
  • When stop/delete is selected, a dialog appears to warn user and confirm action

Volumes Table

While we have the volumes table underneath the notebook servers table, the new kubeflow version in MiniKF has it as an extra menu option in the left sidebar. This table has additional features such as:

• Displaying Access Mode and Storage Class
• Allowing user to browse volume contents: view, download, open, upload, etc.
• Creating a new volume or cloning an existing snapshot right from the volumes table (don't have to create a notebook server first)

[ca-scribner]

@sylus @zachomedia @brendangadd @blairdrummond ^

What is/isn't in 1.2 for the jupyter web app is hella confusing, but Saffa's got a solid summary if anyone needs a good place to start for figuring it out.

[saffaalvi]

v1.1 - Manifest Tags

Compared using the v1.1.0 release commits.
FEB 6 - Delete dead code for releasing and updating images
TAG PRESENT IN:

jupyter/jupyter-web-app/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-gd9be4b9e

jupyter/jupyter-web-app/base_v3/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-gd9be4b9e

stacks/ibm/application/jupyter-web-app/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-gd9be4b9e

FEB 25 - [CentralDashboard][Testing] End-to-end testing added
TAG PRESENT IN:

jupyter/notebook-controller/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/notebook-controller
  newName: gcr.io/kubeflow-images-public/notebook-controller
  newTag: vmaster-gf39279c0

jupyter/notebook-controller/base_v3/kustomization.yaml

• name: gcr.io/kubeflow-images-public/notebook-controller
  newName: gcr.io/kubeflow-images-public/notebook-controller
  newTag: vmaster-gf39279c0

[saffaalvi]

v1.2 - Manifest Tags
AUG 7 - Fix #5111: Trim whitespace from custom images
TAG PRESENT IN:

jupyter/jupyter-web-app/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-g845af298

jupyter/jupyter-web-app/base_v3/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-g845af298

stacks/ibm/application/jupyter-web-app/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/jupyter-web-app
  newName: gcr.io/kubeflow-images-public/jupyter-web-app
  newTag: vmaster-g845af298

SEP 29 - Refactor the JWA backend to utilize common code
TAG PRESENT IN:

jupyter/notebook-controller/base/kustomization.yaml

• name: gcr.io/kubeflow-images-public/notebook-controller
  newName: gcr.io/kubeflow-images-public/notebook-controller
  newTag: vmaster-g6eb007d0

jupyter/notebook-controller/base_v3/kustomization.yaml

• name: gcr.io/kubeflow-images-public/notebook-controller
  newName: gcr.io/kubeflow-images-public/notebook-controller
  newTag: vmaster-g6eb007d0

[wg102]

The manifest released the v1.2.0 tag on November 20th.

As it currently was on November 28th, The back end part – notebook-controller - was updated with the image kubeflow/kubeflow@6eb007d to refactor the JWA backend to utilize common code. On Nov 4th, an auto PR updated the notebook-controller tag to the one mentioned above, vmaster-g6eb007d0 (kubeflow/manifests@0f673cf).

In the release, the latest there was a PR to update some of the images in manifest two weeks ago, aka nov 16th kubeflow/manifests#1645, but they updated with rather old tags. (See details for V1.2 in the comment above)

• AUG 7 - Fix #5111: Trim whitespace from custom images
• Sep 29 - Refactor the JWA backend to utilize common code (#5316)

The front-end crud web app changes have not yet been implemented in the V1.2 release commits since we couldn’t find the commit tag anywhere in the manifest. The latest tag is vmaster-g845af298 which has the commit of AUG 7 - Fix #5111: Trim whitespace from custom images (kubeflow/kubeflow@845af29)
As of now, there are still changes being made to both crud-web-app and jupyter-web-app in the kubeflow/kubeflow repository.

To summarize, v1.2 has the commit that contained the crud-web-app backend using common code, however maybe because it is still being worked on, the frontend for crud web apps has not been updated in the manifest.
The commit to initialize the jupyter web app frontend is kubeflow/kubeflow@89b1395 and was made on Oct 27 in the kubeflow/kubeflow repository but it’s commit tag is nowhere to be found in the manifest.

Things to keep an eye on:

• There is a checklist in this issue: Refactor Jupyter web app to utilize the common code kubeflow/kubeflow#5310 : Which contains the element "Update our manifests to use the refactored web app". We should therefore be able to know when they do update the manifest to CRUD by keeping track of this issue.
• The Kubeflow 1.2 release announcement : Feature Name: Refactor Notebooks Web App
• The Kubeflow v1.2 release issue : Kubeflow v1.2 release kubeflow/kubeflow#5371