Kernel architecture review and polish

[rebornix]

This issue covers the architecture review and polish @DonJayamanne and I are working on. The main goals of this task are

• Finalize how the APIs from src/kernels are being used by the core system (notebooks, interactive-window etc) and 3rd party extensions. There will be restriction and clear separation between internal and external use of the kernel interfaces.
• Internal architecture review and polish of src/kernels component. The kernel component will stay minimal and limited responsibilities without worrying about business logics of how kernels are being used (i.e., how to read variables, how to install dependencies into the kernels, etc)

Here is a list of tasks we will tackle:

• Strict IKernel with notebook document linked in core
  • Strict IKernel with notebook document linked in core. #10817
  • Internal system always create IKernel with NotebookDocument, it's 1-to-1 mapping.
  • 3rd party can create kernels from Uri instead of NotebookDocument
• Conditional notebook controller for cell execution Remove controllers from ThirdParty Kernels #10884
  • 3rd party can create kernels, which don't require a VS Code NotebookController to be created
• Remove explicit knowledge of interactive window from IKernel
  • Unify node/web kernel and remove iw reference in kernel component #10853
  • Both Interactive Window and Debugging feature require special initialization after jupyter session is started. StartupCode can be registered from IW and Debugging other than embedding the logic into IKernel
  • The fix would be similar to how we add @multiInject(ITracebackFormatter)
• Debugging and Debug Variable can probably be moved out Rebornix/move debugger out of kernels #10870
  • Debugging and debug variable provider are used by notebook and interactive window so they probably don't need to live in kernels at all.
• 🏃 KernelFinder and KernelProvider Kernel Finder workflow #11153
• Review misplaced internal components
  • ipywidgets Experiment: move ipywidgets into notebooks #10903
  • port

Tasks

Beta Give feedback

No tasks being tracked yet.

[rebornix]

A gist of our discussion of how the kernel detection and launching works currently

[rebornix]

We have polished and narrowed down the API we expose to the core and 3rd party extensions. The core currently only needs to use IKernelFinder, IKernelProvider, and IKernel. The core can always get a IKernel which is attached to a NotebookDocument and NotebookController while 3rd party extensions can request for headless kernels but those kernels won't be visible to the core system.

In August, we would look into what we can do to improve IKernelFinder and see if we can make it transparent to other components or make it easier to use. Along with that we would have telemetry ready to give better insights into how each kernel type (raw, pyenv, remote) works in overall right now.

Draft interface we discussed offline:

interface KernelManager {
    registerKernelProvider(provider: KernelProvider)
}

interface KernelProvider {
    kernelInfos: KernelConnectionMetadata[];
    onDidChangeKernelInfo: Event<void>;

    getKernelOrKernelFactory(metadata: KernelConnectionMetadata)
}

[DonJayamanne]

Here's my proposal (from my private project)

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import { Kernel, ServerConnection } from '@jupyterlab/services';
import { Disposable, Uri } from 'vscode';

export interface KernelSpec {
    readonly language?: string;
    readonly env?: NodeJS.ProcessEnv;
    readonly metadata?: Record<string, unknown>;
    readonly argv: string[];
    readonly interrupt_mode?: 'message' | 'signal';
}
type BaseKernelConnectionInfo = {
    id: string;
    /**
     * Languages supported by the kernel.
     * First language in the list is considered to be the default & primary language of the kernel.
     * Cannot be empty.
     */
    languages: string[];
    label: string;
    description?: string;
    detail?: string;
    /**
     * Connections with this same value will be grouped together.
     */
    group: string;
    /**
     * Sort order for connections.
     */
    sort: string;
};
export type KernelSpecConnectionInfo = BaseKernelConnectionInfo & {
    resolve: (resource: Uri | undefined) => Promise<KernelSpec>;
};
export type KernelFactoryConnectionInfo = BaseKernelConnectionInfo & {
    create(resource: Uri | undefined, settingsFactory: SettingsFactory): Promise<Kernel.IKernelConnection>;
};
export interface SettingsFactory {
    makeSettings(options?: Partial<ServerConnection.ISettings>): ServerConnection.ISettings;
}

export const KernelRegistry = Symbol('KernelRegistry');
export interface KernelRegistry {
    registerKernel(connection: KernelSpecConnectionInfo | KernelFactoryConnectionInfo): Disposable;
}

I don't think we need onDidChangeKernelnfo
Updated and i think we should discuss the changes and where we want this API/changes to go..

Note: The above API is structured with the intent of exposing them for 3rd party.
We could treat the local kernel finder and remote kernel finders and the like as standalone components in the extension that contribute kernels using the above API...
Lets discuss

[rebornix]

The only task left is simplying kernel finder, which is now tracked separately in #11153.

[DonJayamanne]

Closing this issue for now, as we have other issues to track this work