Pre-proposal: incorporation of jupyter(lab)-lsp

[krassowski]

Pre-Proposal Checklist

• Briefly outlines the proposal
• Suggests a review team (optional)
• Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.)
• Identify a Shepherd to see the process through. (Shepherds are assigned on a round-robin basis from a set of interested engaged volunteers).
• Decide if it’s a JEP according to the criteria listed above. The Shepherd decides if the JEP criteria have been met.

Pre-Proposal

The maintainers of jupyter(lab)-lsp would like to propose its incorporation as an official sub-project of Project Jupyter. We feel this would benefit Jupyter users through better discoverability of advanced interactive computing features, supported by the Language Server Protocol (LSP), but otherwise missing in a user's Jupyter experience.

The key component of the repository, @krassowski/jupyterlab-lsp, offers Jupyter users an expanded subset of features described by the LSP as an extension to JupyterLab. These features include refinements of existing Jupyter interactive computing features, such as completion and introspection, as well as new Jupyter features such as linting, reference linking, and symbol renaming. It is supported by jupyter-lsp, a Language Server- and Jupyter Client-agnostic extension of the Jupyter Notebook Server (for the 0.x line) and Jupyter Server (for the 1.x). We will discuss the architecture and engineering process of maintaining these components at greater length, leveraging a good deal of the user and developer documentation.

Much like Jupyter Kernel Messaging, LSP provides a language-agnostic, JSON-compatible description for multiple clients to integrate with any number of language implementations. Unlike Kernel Messaging, the focus is on precise definition of the many facets of static analysis and code transformation, with nearly four times the number of messages of the Jupyter specification. We will discuss the opportunities and challenges of this complexity for users and maintainers of Jupyter Clients, Kernels, and related tools.

Branding

We invite discussion on an appropriate Jupyter branding/organization approach, ranging from:

• a single move of the @krassowski/jupyterlab-lsp monorepo to @jupyter/lsp
• a new, Jupyter co-branded GitHub organization (e.g. jupyter-lsp)

Kernel integration

In order for jupyterlab-lsp to be able to connect to the appropriate Language Server in tandem with a Kernel, it requires that kernels properly implemenet KernelInfo from version 5.0 of Jupyter Messaging specification, in particular the field language_info with the following fields present:

• mimetype
• file_extension
• name

Therefore, no changes to the specification are required. Though some future specification refinements could be discussed as separate JEPs.

In continuous integration, we keep under test Kernels and Language Servers for all three of the Jupyter-name forming languages:

• Julia: IJulia using LanguageServer.jl,
• Python: IPython using pyls or jedi-language-server,
• R: IRKernel using R languageserver

Other kernels also confirmed to work straightaway (Bash) or after minimal user configuration (Scala) in presence of appropriate Language Servers. When accounting for language servers that only support the file editor but not notebooks, a total of 13 language servers is known to work with the jupyterlab-lsp.

Integration with alternative frontends might be possible in the future; as a proof-of-concept we were able to activate most of the LSP functionality in the jupyterlab-classic (a prototype re-implementation of the classic interface using JupyterLab components).

Interestingly, the extension was adopted as a basis for other notable JupyterLab extensions, and it was used to prototype interesting novel concepts in academic context.

Magics and polyglot support

We also support the magics in IPython, allowing for polyglot experience when using cell magics. For the detailed discussions on magics integration please refer to our discussion on code overrides (which is our term for generalisation of magics, shell expressions and pinfo ?/?? syntax).

Outlook towards notebook-specific IDE features

The team envisions a support for the notebook specific IDE features to be included in the future. This might involve upstream changes to the LSP specification, or custom jupyter-specific extension. Please see our community discussion for more details.

Minimal implementation details

• jupyterlab-lsp implements integration with JupyterLab and its default CodeMirror editor; the design allows for incorporation of additional editors in the future, but large portions of code are CodeMirror 5-specific; it communicates with jupyter-lsp to send/recieve messages from language servers and with JupyterLab to get information about currrent kernel
• jupyter-lsp uses jupyter_server, tornado, sockets and IO streams for communication with servers and with frontend; it is not aware of a kernel's existence
• we use GitHub CI running pytest, jest, and robot (Selenium) test; the integration/acceptance tests for the bulk of the frontend testing; we test on Windows, Linux and MacOS, supporting thee Python versions at a time for jupyter-lsp and doing tests with actual language servers

Previous discussions:

Please refer to:

• Advise on closer jupyterlab-lsp integration with Jupyter(Lab) jupyterlab/frontends-team-compass#67 for the discussion held within the JupyterLab team.
• Add language server (LSP) support in notebook server and jupyterlab-monaco #26 for previous (stale) JEP on LSP adoption; this proposal superseds that JEP; importantly jupyter(lab)-lsp does not require any specific editor, whereas the previous JEP specifically mentions Monaco.

[krassowski]

We invite discussion on an appropriate Jupyter branding/organization approach, ranging from:

• a single move of the @krassowski/jupyterlab-lsp monorepo to @jupyter/lsp
• a new, Jupyter co-branded GitHub organization (e.g. jupyter-lsp)

A new organization could be named jupyter-code-assistance:

• it could welcome projects such as Julynter should they (or future alternatives) decide that its the best place for them,
• we may want to use a generic term over a technical LSP abbreviation as this may not be well known by the users,
• it could also host hypothetical future Jupyter-optimized servers, such as ipython-language-server, ijulia-language-server, etc - should we decide that there is a need for forks of language server that cater to interactive computing extensions of the most popular Jupyter languages (I imagine those would be mostly soft and friendly forks that would contribute a lot of the features upstream, but maintain a build optimized for the Jupyter environment)

[krassowski]

It seems that another term is sometimes used: content assistance, which signifies that these features are not limited to actual code. Indeed the LSP extension already works well with Markdown, providing spellcheck and linting among others. However, I think that code-assistance is more widely understood term.

There is a precedent for similar goal-oriented organizations in the Python real: the Python Code Quality Authority.

Edit: FYI I brought it up as I would like to hear some feedback before sending a proposal in the PR. The work on the proposal is in progress, realistically two-three weekends away from completion.

[krassowski]

Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.)

It should be a JEP because the adoption of analogous Debugger Protocol was a JEP #47

[lresende]

We invite discussion on an appropriate Jupyter branding/organization approach, ranging from:

• a single move of the @krassowski/jupyterlab-lsp monorepo to @jupyter/lsp
• a new, Jupyter co-branded GitHub organization (e.g. jupyter-lsp)

A new organization could be named jupyter-code-assistance:

• it could welcome projects such as Julynter should they (or future alternatives) decide that its the best place for them,
• we may want to use a generic term over a technical LSP abbreviation as this may not be well known by the users,
• it could also host hypothetical future Jupyter-optimized servers, such as ipython-language-server, ijulia-language-server, etc - should we decide that there is a need for forks of language server that cater to interactive computing extensions of the most popular Jupyter languages (I imagine those would be mostly soft and friendly forks that would contribute a lot of the features upstream, but maintain a build optimized for the Jupyter environment)

We are making this much harder than necessary. IMHO, we just need to get the JupyterLab project to accept this as one of the extensions on the jupyterlab org. Anything else (e.g. new org, etc) would just contribute to the fragmentation of the community.

[krassowski]

Thank you, I agree with the concern. To reiterate, there are a two major reasons why making a dedicated GitHub organization makes sense:

• we would like to have multiple repositories and we are hitting limits of what is possible with a monorepo (e.g. we would like have different demo binders for various languages); we would like to be able to host notebook-specific extensions of existing language servers as separate repositories
• we are trying to develop LSP integration which can be shared between all frontends (JupyterLab, Notebook, ntreact, etc); we already have jupyter-lsp which is not specific to JupyterLab at all; this was raised in previous discussion on JupyterLab team-compass: Advise on closer jupyterlab-lsp integration with Jupyter(Lab) jupyterlab/frontends-team-compass#67 (comment)

We can still have multiple repositories when inside of @jupyterlab/ organization but this is getting difficult to navigate with lumino, nbclassic and 52 other repositories already living there. We are missing out on discoverability either way (sub-repositories not discoverable from within org as too many repos in @jupyterlab/, or not discoverable from @jupyterlab/ because in a different org but sub-repositories discoverable within the new org).

[bollwyvl]

Yeah, I'm not super bothered one way or the other in terms of the org it gets homed to, but do think this will generate a fair number of repos, and eventually have implications for many clients and different specs, not just JupyterLab:

For example:

• if we go down the comm approach (which i think we should, eventually) we might want to formally define its comm message signature, which would handle both the lifecycle (some of the servers die... a lot...) as well as the content of the actual messages
• a notebook could embed LSIF (e.g. pre-computed responses to hover, signature, go-to-definition, etc) of its code (and or markdown) content within its metadata (or as an output), and nbconvert could annotate the resulting output
• as mentioned in the doc, CodeMirror 6 (which i think most folk want to see adopted for perf/typing/a11y) has stated that first-party LSP isn't gonna happen... maybe we build it as a non-Jupyter-depending (but probably @jupyter-*-branded) extension that leaves us enough surface to integrate multiple clients