Draft proposal for new kernel providers mechanism and associated changes #45
Conversation
|
Thanks for opening this. |
|
I like the idea. In particular because the "putting environment name in the kernel name" pattern is getting more and more use. I think the notebook extensions that offer this are used by people who want an easy way of switching between environments. The problem occurs when they then send their notebook to someone else. Often it takes a long time (if they haven't run into this problem before) for both sides to figure out that this is what happened. One question: using entrypoints as the discovery mechanism is cool. How would this work if neither your frontend nor your notebook server are written in Python? I am not sure this is a problem right now but I could imagine a world where a UI is written in JS, running a JS kernel and no Python is ever involved (or replace JS with some other cool new language not yet invented). Is there a nice way for them to also use this mechanism? |
|
@betatim this is effectively a Python-Jupyter-client feature for new kernel discovery methods/dynamic kernel creation (i.e. env-kernels, remote kernels, etc.). The existing 'kernelspec-file' mechanism that is easily shared across implementations continues to exist and be shared by all frontends, and is one such discovery mechanism. I'd love to have a comment from nteract folks like @mpacer or @rgbkrk on whether adding features like this that are Python-frontend-specific is seen as an issue from their side. One mitigation could be a Python CLI for listing/launching available kernels. |
|
Should we also broadcast this a bit wider to see what reactions there are from the Julia and R spaces? Which other ecosystems do we know of that have both active Jupyter-using communities and enough "environment management" tools and traditions for this complexity to arise? If there are any, perhaps we can try to ping their channels for input? |
|
I approve this JEP. We were already looking at Thomas' |
It does essentially tie part of the ecosystem to Python. Of course, it would be possible to have interfaces (command line, HTTP, ZMQ, etc.) which would allow non-Python code to use this mechanism. Or, as Min mentions, you could reimplement particular kernel discovery mechanisms in another language (or hardcode a way to launch a specific kind of kernel). But to fully support this kernel discovery framework, you would have to have some Python code running. I would argue that it's easier to define language-agnostic interfaces for discovering & launching kernels around the Python code in |
|
Would the intent for kernels with the same language but differing kernel implementations be that you give the notebook a different language name in the metadata? e.g. with my kernel for environment X can only run in environment X but in X there's multiple extensions of a base language kernel -- should I name my kernel language Thoughts? |
I think entrypoint registration is a nice feature but forcing python on all kernel management seems constraining. I do see the argument for simplification, but since everything else is language agnostic in jupyter contracts I think this would add friction to low-level developers. What's stopping kernel discovering by scanning shared filesystem paths as an alternative as a crude way to support kernel command discovery? |
|
I think it'd be beneficial to document the boundaries of usage with examples, with the associated metadata representations, when this moves forward. Especially for cases where it's expected that the stack developer supporting the pattern needs to manage their business rules around how to couple a notebook to a kernel. The risk here is having notebooks accidentally run with the wrong kernel and not having visibility into what's wrong or why it's occurring. Independent of how associations are made, we should probably include a path for better visibility into how a kernel was chosen and how to fix when a wrong kernel is being picked by the ecosystem. |
|
Hoping those comments can be constructive for improving the doc. I'm not opposed to the idea in general but those things popped out to me as I was reading which probably have been thought through but I didn't get from the proposal document. |
|
Following @jasongrout's point of order in #44, I updated the top comment to include a similar process for voting from the steering council. |
No! That's going back to what we have now, but calling the field 'language name' instead of 'kernel name', so it's more confusing. The idea is that kernel names do not have a global shared meaning. E.g. I might create a conda environment (and associated kernel) called I haven't worked out in detail what should happen instead - I'm hoping to make it flexible enough to allow for experimentation in how kernels are chosen for existing notebooks. E.g. maybe a notebook extension could embed dependencies in the metadata, and then when opening the notebook, find or create an environment that has the necessary dependencies. Maybe another extension could check for Docker image IDs. This bit is definitely the most hand-wavy part of the proposal - it's an unsolved problem which I'm hoping people will play around and find good answers for. But describing kernels as 'languages' is precisely the mistake I'm trying to get away from.
We've got something like this at the moment with kernelspecs. Of course, you could have another layer of filesystem-based discovery like this to launch kernel providers, which then launch kernels. But how does your kernel provider then give your application back something like what JKM calls a manager, which exposes operations like sending signals and waiting for the kernel process to terminate? I think inter-language interop here is something that sounds nice, but doesn't actually solve many problems in practice. If you allow kernel providers in any language, applications have to use them as subprocesses in general. But if that's the case, specifying that they're in Python is no extra burden. (I'll read the further responses soon) |
I've been thinking about this lately and wondering if the provider of a given kernel-type could "advertise" the kernel-type's capabilities. This idea is similar to the dependencies that Thomas speaks of but more about the environment in the kernel (notebook code) will run rather than what the code in the notebook does. The primary capability being 'language_name' of course, but if these advertised capabilities of the selected kernel were persisted in the notebook file, then upon its transfer to another system, a selection process could occur if the language/provider/name tuple isn't found. Since kernel-types are not aware of the actual code used in the notebook, they can only provide an idea of what they're capable of. However, assuming the kernel-type was a good match when the notebook ran initially, then, on a different system, that notebook could be more easily paired to a kernel-type that matches language and most, if not all, advertised capabilities with its persisted capabilities. In addition, I wouldn't view things like 'kubernetes' or 'docker' or 'YARN' so much as capabilities. These, IMO, are more about the deployment model and the notebook will likely not contain code specific to these entities. I view capabilities as things related to the executable code that resides in the notebook: 'python', 'tensorflow', 'numpy', 'spark', 'gpu', 'memory' would be examples of capabilities available to a kernel-type of language 'python'. This capabilities discussion is probably more related to JEP #46. |
In other words, JKM ensures a backwards compatibility. Any kernel written in any language having today valid json file at the correct place will be discovered and available for usage. The kernelspecs providers does not need the kernel implementation to provide any additional python entrypoints. Based on this, my understanding is that this proposals supports any kernel language (JS, R, Julia...) just like today. |
|
Thanks for opening this, @takluyver! I've been lurking on the kernel management work you've been doing over the past few months since I've seen this problem (ambiguity of kernelspecs) affect quite a few people at the organization I work out.
The nteract core state model currently does something similar to this. Each kernelspec has a unique reference associated with it that can be used for correlations between other data types in the state model. For example, a kernelspec with ref A can be associated to kernels B and C and content D. I've found this model to be a nifty way to reason about kernelsepcs and have seen other developers building notebook UIs benefit from this abstraction.
I like the model that @kevin-bates presented of associating capabilities (which in my mind I am mapping as dependencies) with language types within the new kernel type definition. Although I'm not sure how it'll work out practically with dependency versions and such.
Ideally, the new kernel type model would provide a simple heuristic for associating a notebook with its matching kernel. I've had to write some pretty gnarly logic to resolve by kernelspec name, then codemirror mode, then display name, and a variety of other things to try and get something that would reliably match kernelspecs in the past.
I'll chime in on this, @minrk. A CLI is fine, there are packages in the nteract ecosystem that are essentially NodeJS wrappers around the Jupyter CLI. Ideally though, I would try to go for what @takluyver proposed and have a common standard that folks can implement in Python, NodeJS, Go, C#, and so on. I'd be happy to help validate if this is language agnostic enough by building a NodeJS implementation. |
Same thinking here. |
There was a problem hiding this comment.
Thanks for replying to the comments @takluyver ! And thanks for pushing a change forward. I think enough folks have chimed in and the concerns I raised are mostly covered or thought through.
|
This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there: https://discourse.jupyter.org/t/sustainability-of-the-ipynb-nbformat-document-format/3051/4 |
|
I'm approving this in the interest of moving things forward. I am glad that this proposal embraces backward compatibility with the current system. I have concerns, for example, about the added complexity, especially for new users. The phrase "It was a mistake to hide this simple concept behind a confusing 'user friendly' veneer." elevates the particular kind of advanced user who has multiple environments as a primary target user. But every Jupyter user started with one environment, and legion of them continue to use exactly one environment. So there's certainly a trade off with how visible this machinery should or could be in the user interfaces. |
|
Trying to move this forwards again: the remaining people who haven't voted are @jasongrout @jhamrick @mpacer and @Ruv7 . Did we set a quorum level to declare a vote like this complete with <100% of people voting?
I have concerns about complexity as well, but I was trying to say that the previous design tried to be 'user friendly' in a very superficial way, and ends up being user unfriendly in a much more important way. We focused on hiding underscores etc. behind a 'display name', but it's easy to end up with multiple kernels with the same display name, which is far more confusing. If you only ever have one kernel available, you can't run into that problem. But I don't think I'm proposing anything that would make the experience worse in that case. |
|
@MSeal FYI. @takluyver I'm going to work off of @choldraf's suggestion re: acceptance. Seems reasonable to do so here too. Feel free to cast your vote on #62 too 😄 |
|
I'm still not 100% sure what will need to change in jupyter_client / nbclient / notebook_server to support the concept. Replacing jupyter_client with jupyter_kernel_mgmt is non-trivial at this point and changing that is very likely to cause a host of issues for nbconvert, nbclient, and papermill -- though @kevin-bates knows this well I believe. I also would like to know what the spec changes would be so it's clear when implementing support in different layers how the nbformat and kernel communication would be different. e.g. Where does the kernel id go? It's noted it should be unique but not at what level of enforcement of uniqueness should be held. If a kernel id doesn't exist in a system, it's implied we should fail but not explicitly said. Are we committing to that? The sharing a notebook between users with slightly different environments will start failing if we do that. I'm not sure how to communicate cleanly to a novice user how to fix this in development their flow. Not trying to say put on the brakes on the proposal, but as one of the maintainers of a lot of things that are impacted I worry about things braking because these points aren't clear -- to me at least -- and I'm not going to be able to spend much time in the next couple months helping with understanding / applying said change. Long winded, but what's the expectation beyond moving those repositories into jupyter org upon merging this proposal? Is it worth having a walk through on how the higher level abstractions need to change to adopt the proposal post-merge? |
|
To be clear I'm for interface improvements here and agree with the basis of the argument for the proposal. I was hoping more of these things would pop out from discussion in this thread and didn't want to only be prompting on potential problems, but if it's going to be merged I'll maybe ask for some more clarity on the implementation half of the proposal (can be in a different thread or context if needed). |
|
@takluyver - thank you for kicking the proposal! @MSeal, thanks for your additional questions and concerns. I've taken the liberty to answer these as best as I can. Thomas, please feel free to correct me or comment as you see fit.
There are no plans to switch to this mechanism in the notebook server. It will continue to use jupyter_client for its kernel management and communication layer. Jupyter server, on the other hand, does have plans to adopt the kernel provider framework in its 2.0 release.
I'm not familiar with how those applications instantiate and interact with jupyter_client's KernelManager, but, you're right, it may be non-trivial. Updating the notebook (jupyter) server certainly was, but the gain in flexibility is significant. I think as applications move to adopt kernel providers, we'll find the need to simplify things. For example, I wouldn't mind seeing the new We will likely need to support two forms of kernel management during these times since this kind of change cannot be made overnight. The backward compatibility support, for the most part, is sufficient, although those notebooks reliant on custom KernelManagers and/or KernelSpecManagers will likely be prompted for kernel changes as their existing kernelspecs will not be found or work correctly. I call out some areas of compatibility concerns in the jupyter_server PR referenced above.
By kernel id are you referring to the Kernel Type ID that will prefix the kernel name upon return from the server? I'm assuming so. In that case, the kernel_name field used today will contain a prefix with a Likewise, applications that have not adopted kernel providers and encounter a notebook last used via a Provider-based scheme will need to handle the prefixed kernel_name. This is probably something we could place into jupyter_client's
This is a good question. The current thought is that uniqueness will be at the package level (i.e., let the package-registry be the registry).
I believe this should be up to the application. This isn't necessarily anything the JKM framework would provide or decide. Some applications may decide to best-match the language, proliferating today's behavior. Others may prompt the user to install Kernel Provider XYZ or change the kernel themselves (caveat emptor).
This is the age-old question in this space (from the brief history I have witnessed). I believe Kernel Providers get us closer to a resolution because they bring the kernel lifecycle-management (which includes construction of the environment in which the kernel runs) into the picture. KPs seem like a step forward, although, I agree, they don't solve everything.
I think if we get another application or two migrated, we'll see the necessary patterns, then we can apply any necessary refatoring to make migration easier, and, yes, a walk-through would be good at that point or during. |
|
@takluyver I'm for the change but I'm a bit concerned about increasing confusion about kernels which are already confusing for many users. Was there ever any discussion about using the name "Environment Provider" instead of "Kernel Provider"? |
|
Thanks for the clarifying details @kevin-bates ! If it's alright I might respond inline with follow-up questions / things that might want to be dug into as a Q&A here or as a post-merge effort.
Thanks for the clearer description. Can some of these details to the JEP so it's clearer?
So should the prefix default to being the package name? This would be nice because you would know the package is unique within a distribution. Though packages could have multiple kernels, so is there guidance in that situation? Or am I misunderstanding the naming convention?
Understood. I just want to avoid fragmentation with no clear plan around convergence. It's already pretty hard to debug and explain what's wrong at the kernel interface layer in issues and having two potential paths for an extended time will be a maintenance burden. I think if we had a partial implementation running for nbclient to use this proposal it'd give a better idea of the gaps to convergence. Would / could we establish a timetable for deprecation of the older pattern?
Agreed. That might be a hard sell as custom kernels in the wild will not be happy with us.
That will mean inconsistent behavior between applications in a critical path of execution. For me I envision a lot of "it works this way in XYZ application, why doesn't papermill do the same?" with no good resolution because if we make it work like XYZ then it'll be different than application ABC's behavior. Am I being overly concerned here or is it clearer to other's how the difference in execution will be handled well?
I personally think that we need clear instructions or tooling changes for kernel lifecycle-management if we change how it works today and that without really solid how-to's and simple operations we'll make it impossible for non-experts to resolve kernel issues when sharing notebooks. I recognize this might be too large an ask for the JEP itself, but denoting it as a requirement to be worked out before services adopt KPs might be warranted. As an example I don't really understand how a .NET or Scala kernel will be available to the provider or how to ensure it is available at runtime in a consistent manner. I don't believe telling them they have to write a parallel python package is going to go over well if that's the plan (e.g. recent Julia kernel conversations) and if it is what we're committing to it needs to be Super clear for a non-python developer how to walk through that imo. Hopefully this is a useful set of discussion points for everyone else as well, but thank you for engaging on the questions / concerns for my sake at least. I believe it will help a lot down the road for application support. |
|
Thanks for your responses and questions. My responses follow:
I'm not sure what else to add. Kernel Type ID seems fairly well-defined. Perhaps a comment stating something to the effect: With this proposal, the kernel_name returned in the discovered kernel specification would reflect the Kernel Type ID rather than the kernel name itself. Would that help?
I think you're conflating kernel implementations with providers and it might help to think about this as @willingc suggests, environment providers. A given Kernel Provider will not necessarily be tied to a particluar kernel. For example, a Kubernetes provider would be responsible for managing the kernel lifecycle of various kernels within a k8s cluster. This single provider might return discovery results across several kernel implementations.
I think this would be a good idea, although I think such a timetable should include input from various stakeholders - i.e., any application that currently operates on notebook files and uses jupyter_client for its kernel management. After thinking about things, I think we might be able to buy ourselves more time by NOT prefixing kernel names with
Hmm - I think you're mixing kernel implementation again here. Any custom kernels that are discovered and managed by the default framework today will continue working with the kernel provider framework tomorrow. Only applications reliant on overriding
This is a valid point for the (presumably) rare times a given provider, outside the built-in default, does not exist. Frankly, I think it will be quite rare to run into a Kernel Type ID that includes a prefix other than
Again, for the 90% case, nothing changes. For a kernel provider implementation that is specific to a particular kernel, I would argue its up to that provider's installation and/or documentation to address necessary requirements and/or provide tooling to configure its resources so its discovery framework can produce the appropriate specifications.
I completely agree Matt and appreciate your interest, concerns, and questions. Thank you. |
|
I really appreciate the patient responses here.
I think I get that but I'm unclear what their relationship will become between kernels, providers, and other services and so am probing the edges. The overall role everything will play post this change in relation to each service / library is what's confusing to me. It might be helpful to have the relation between libraries and tools post acceptance flushed out and to know what would have to change for existing kernel authors and library maintainers outside of jupyter_server would need to do for full adoption. e.g. should the default implementation provide all the existing kernels as-is from the provider to avoid compatibility concerns. If there were jupyter_server meetups that dug into this in more depth, I might have been missing some context and intent here. |
No worries. I'll start with a question: What's the relationship between kernels and jupyter_client's I'd also like to point out that a primary intention of this proposal is to permit multiple
If those kernels "advertise" their availability via
There are weekly jupyter_server dev meetings. We've only had discussions regarding kernel providers on occasion. Anyone is welcome to attend to discuss any related topic. |
That helps describe default behavior, thank you.
Didn't mean any negativity about that meeting therein if that was implied, I just literally hadn't attended them and was probably missing context as a result. |
I'm sorry Matt if my response was taken in that vein. Reading it back now, I can understand how that may have come across. It was meant to be informative for yourself and others and nothing more. I should have prefaced it with something like, "In case you're interested...", but, honestly was a bit haste to end my day. Have a good evening. |
|
I had the same feeling about my earlier statement and didn't assume any ill intent on your response. You have a good evening too :) |
I don't think there was. Environments are a convenient starting point to explain this in a Python context - we're used to creating multiple environments with conda or virtualenv, and that's an easy way to end up with multiple kernel types for one language. But you could also have a kernel type for Python running in a Docker container from image XYZ, or an emulator for a different CPU, or on a specific remote machine. You could call all of these 'environments', but I'm not sure people would readily make that leap from the name. |
|
@takluyver Thanks. I see the point for the use cases beyond Python. Maybe |
|
There's certainly scope for confusion around the term 'kernel', but I'm not sure what confusion is avoided by adding the word 'execution'. I've tried in this proposal to distinguish 'kernel types' (e.g. Python running in this environment) from 'kernel instances' (e.g. process 1234 listening on these ports). The 'kernel providers' described here could be seen as kernel type providers in the first instance - e.g. a provider for conda environments would offer new kernel types, but kernel instances would be managed as subprocesses by the same code as for the @MSeal :
This is a very real concern. Given how long this has been hanging around already, I can't realistically claim that I'm ready to swiftly integrate it with the ecosystem. There were proof of concept PRs for projects like nbconvert, but these have long since gone stale. And given how hard I've found it to explain even to people involved in Jupyter development, confusion in the wider community is all but inevitable. Perhaps it's more sensible to let this drop and look for ways to improve the situation more gradually. |
To be completely honest, I'm feeling the same way (and have been for a while) but felt things needed to be supported given the time spent. I wasn't aware of how prevalent Should this proposal be retracted, I do have ideas for how we could implement different "execution environments" by simply abstracting the process (Popen) layer within |
That does sound promising and a smaller incremental change to the ecosystem |
|
I still think it would be valuable to have a more flexible way to discover kernel types - e.g. allowing something like nb_conda_kernels as an addition to standard kernel discovery, rather than a replacement of it. But if anything like that is going to happen, it will need someone with a decent overview of the Jupyter world to design and push it.
I did initially try to do this as a refinement of Anyway, sorry to waste everyone's time by pushing for votes on this. |
I completely agree.
Your insights are never a waste. I value all that you bring to Jupyter. ❤️ |
|
Some additional comments on some of the patterns I am seeing in how the kernel/kernelspec abstraction related to "environments" (either conda/pip envs, or docker containers):
|
|
As requested on the steering-council mailing list.
I've restructured what I'm saying to make it clearer that moving the two repositories into the Jupyter org is only one part of this.
Votes for approval: