How to use with notebook

[echarles]

Hi, Great to see this!

How can I test it with the notebook? Should I first apply jupyter/notebook#4170 (which btw has merge issues)?

PS: Any plan to create the yarn_kernel_provider implementation?

[kevin-bates]

Hi Eric - please keep in mind this is completely experimental stuff and is subject to change (which includes its complete removal) at any time - caveat emptor. 😄

Yes - that is the Notebook PR on which to base your testing. Given the branch is 2+ years old and completely reworks kernel management, it will certainly have merge issues. In addition, I've had to make changes to get remote kernels working but haven't conveyed those back to @takluyver.

I'll try to add further instructions for setting up an environment. It will consist of manual branch pulls , builds, and installs. I've been able to run kernels in YARN Cluster mode so far - looking at Kubernetes now. There are still issues with restarts and configuration that need to be worked out.

Not sure what your yarn_kernel_provider question is about. The yarn provider implementation was posted 5 days ago in https://github.com/gateway-experiments/yarn_kernel_provider which is another thing that might change (i.e., one repo per provider + the base).

[echarles]

Happy to cook, pull and merge branches when you provide the receipt.
Oops, I was meaning docker_kernel_provider (upon yarn and k8s ones)

[kevin-bates]

Ok - once kubernetes is in a working state, docker should just follow suit. However, before we include more than k8s, yarn (and the remote base), I want to take a step back and make sure this "repo layout" and deployment model is what we really want.

Also keep in mind that all the configuration support for k8s and docker (helm charts, compose scripts, etc.) are geared toward EG being the server. Those will require rework to have Notebook be the server and that may prove challenging.

[echarles]

Happy to test on K8S or YARN with EG as server.

[kevin-bates]

Thanks. EG will not be usable with kernel providers for some time. I suspect you meant to say with Notebook as a server.

[echarles]

yes, notebook as a server. sorry.

[kevin-bates]

Hi Eric - I really apologize for the delay here (been side-tracked lately). I've managed to get back to this stuff this week. I'm currently in the process of getting the k8s kernels off the ground. I'm running into issues with the base notebook branch (that we need to use) - which is preventing getting to the crux of the kubernetes stuff.

At any rate, my plan is get k8s kernels minimally working. This should validate the ContainerKernelLIfecycleManager on which the Docker lifecycle managers also depend (we've changed the 'process proxies' to 'kernel lifecycle managers' (after all, what's a few more characters to type 😄). Then, if you're okay with things, it would be great if you could follow the k8s lead to get the docker stuff working!

I also owe some instructions for getting things going. I'd prefer to wait until one of the PRs is merged on the jupyter_kernel_mgmt repo that allows us to derive from the KernelSpecProvider rather than implement our own 'find' logic. Once that is merged, we'll make a release of that module - which will simplify deployments. That said, I'd be happy to share which 'branches' are necessary if you're chomping at the bit.

There are many issues to work out in this 'provider world' and I'd have to say the jury is still out on this (IMHO).

I hope that helps.

[echarles]

@kevin-bates Thx for the updates and the hard road you are following to lead us to a distributed and managed kernel world. Ping me if I can help to test or solve outstanding issues.

[kevin-bates]

Hi Eric - thank you for your patience. I was able to get a "minimally viable" kubernetes instance running. I also have created a docker_kernel_provider repo with the basic substitutions in place (which is essentially all I needed for k8s) but have not tried anything in a Docker/DockerSwarm env. I'm hoping you might be able to try things out.

I also decided that, while we're still checking things out, a "fake" release might be the best way to convey setup instructions. This allows me to attach pre-built files to the release so others don't need to pull branches, wait for PRs to be merged, etc.

You can find this 'interim-dev' release here.

[echarles]

Thx @kevin-bates. I have been through the interim-dev release instructions and tried the on-prem (non container) setup: notebook, juptyer_kernel_mgmt, remote_kernel_provider and docker concrete implementation of remote_kernel_provider.

I deployed the kernelspec and tried the Docker Kernel. This fails with following exception.

I 18:22:49.596 NotebookApp] DockerSwarmKernelLifecycleManager: kernel launched. Kernel image: elyra/kernel-py:2.0.0rc2, KernelID: 66edaa4f-72e3-4a35-a0a8-62eff718fdf9, cmd: '['/opt/datalayer/opt/miniconda3/envs/datalayer/bin/python', '/usr/local/share/jupyter/kernels/py/scripts/launch_docker.py', '--RemoteProcessProxy.kernel-id', '66edaa4f-72e3-4a35-a0a8-62eff718fdf9', '--RemoteProcessProxy.response-address', '192.168.1.23:55374']'
Traceback (most recent call last):
  File "/usr/local/share/jupyter/kernels/py/scripts/launch_docker.py", line 114, in <module>
    launch_docker_kernel(kernel_id, response_addr, spark_context_init_mode)
  File "/usr/local/share/jupyter/kernels/py/scripts/launch_docker.py", line 43, in launch_docker_kernel
    param_env.pop('PATH')  # Let the image PATH be used.  Since this is relative to images, we're probably safe.
KeyError: 'PATH'

Comments on the on-prem-start-notebook.sh:

• It enforces gateway user - I removed that?
• There are configs for YARN. Maybe you tested it with YARN and it would be easier that I first try with YARN instead of Docker?
• There are also EG_* env variables set - I guess in the on-prem case they don't come in the picture?

Then I deployed the K8S Spec on minikube. I get running nb-kernel-mgmt and kernel-image-puller which run the notebooks. When I run python with a Python on Kubernetes kernel, a Pod for the kernel is launched and it works fine.

I wonder how to deploy a notebook outside of K8S which would act as an Enterprise Gateway.

[echarles]

@kevin-bates To complement previous comment, I wanted first to install the kubernetes_kernel_provider as concrete implementation in the on-prem scenario, but decided not to do that as the K8S nb_kernel_mgmt pod is reachable in my case on http://192.168.64.10:31240 (minikube) and I didn't see how to set this in the launch script... Maybe EG_REMOTE_HOSTS but how to set the port?

[kevin-bates]

Hi Eric - thanks for the update.

The on-prem stuff only applies to the YARN support. I've minimally tested with YARN and Kubernetes. For K8s, the notebook server runs in k8s as well. Same will be the case for Docker.

We do not support container envs using on-prem kernels or vice versa.

Regarding your comments:

It enforces gateway user - I removed that?

That's fine. The startup script was pulled (obviously) from my EG YARN env.

There are configs for YARN. Maybe you tested it with YARN and it would be easier that I first try with YARN instead of Docker?

As noted above, I "tested" YARN using on-prem configs and K8s using the full container configs. If you have a YARN env to hit, it might be a good exercise to ensure you see the same YARN behaviors, but that config is not applicable to the container configs. If you choose to checkout on-prem YARN, you might want to validate YARN using EG first.

There are also EG_* env variables set - I guess in the on-prem case they don't come in the picture?

Some might come into play. It depends on where those variables get used in EG. However, some apply to DistributedProcessProxy - which hasn't (and may never) gotten implemented in kernel provider land yet.

I wonder how to deploy a notebook outside of K8S which would act as an Enterprise Gateway.

Not sure what you mean. If you're saying you'd like to take a Notebook server and run it in headless mode, where a "front-end" configured with NB2KG hits that server... you won't be able to do that for a couple reasons. 1, the token management stuff will get in the way. 2. Kernel Gateway has handlers that look for an 'env' stanza in the kernel startup request. This functionality doesn't exist in Notebook server.

The play is to plumb the new jupyter_server with this capability. At that time, we'd likely add support for parameterized kernels where the kernel parameters also appear in a stanza in the kernel startup request.

[kevin-bates]

EG_REMOTE_HOSTS only apply to DistributedProcessProxy. If you're describing how to access the web server to run a kernel, yeah, you either need to setup an ingress or locate the IP and port (which I suspect is what you list in your comment) and hit that url for accessing the notebook.

[echarles]

The on-prem stuff only applies to the YARN support.

Good to know. So it is like Notebook_Server_On_Host <---> Kernel_Manager_On_YARN - See the // for K8S in the next paragraph...

Will further work on validation on-prem with YARN, with prelimary validation of my setup with EG on YARN.

Not sure what you mean. If you're saying you'd like to take a Notebook server and run it in headless mode, where a "front-end" configured with NB2KG hits that server... you won't be able to do that for a couple reasons. 1, the token management stuff will get in the way. 2. Kernel Gateway has handlers that look for an 'env' stanza in the kernel startup request. This functionality doesn't exist in Notebook server. The play is to plumb the new jupyter_server with this capability. At that time, we'd likely add support for parameterized kernels where the kernel parameters also appear in a stanza in the kernel startup request.

I was thinking to Notebook_Server_On_Host <---> Kernel_Manager_On_K8S (// to YARN here above) - The token security would remain local and I would expect the manager on K8S to have all what is needed.

[kevin-bates]

I'm still not sure I'm understanding what you're completely driving at.

So you're Kernel_manager_On_xxx is the headless server? If so, the kernel manager needs a "hosting application" since it doesn't know how to respond to the REST calls that get forwarded from Notebook_Server_On_Host.

With your original question, I thought you wanted something like EG running OUTSIDE of k8s but launches kernels INSIDE k8s - which we don't support. The entity that launches the kernels must also be within the same "network".

If that's still not what you're getting at, I'd be happy to setup a webex or something. Feel free to ping me at kbates4@gmail.com or on our gitter channel/room.