[WIP] jupyter editor including extensions

[costrouc]

This WIP will be working on #49807 and valuable input from @MMesch. The idea is to bundle extend the jupyter editor with `extensions support. This requires several steps.

• include extensions (npm packages) and node packages (15 included for now)
• link extensions with jupyter labextension link <derivation> and build with webpack.

Linking extensions allows for each npm package to be included separately. --app-dir and JUPYTERLAB_DIR control the directory that jupyterlab uses.

Motivation for this change

See #49807

Things done

See above.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[costrouc]

@MMesch currently this PR allows you to nix-shell -I nixpkgs=<path to nixpkgs> -p nodePackages."@jupyterlab/toc" and you have the extension with all dependencies in node_modules. Would be happy to get additional feedback if you have time to experiment with labextension link

[MMesch]

ok. I'll test it tomorrow

[MMesch]

I bumped the jupyterlab version:

MMesch@2dc2700

afterwards this dirty shell seems to work!

{
  pkgs ? import ./nixpkgs {}
}:

pkgs.stdenv.mkDerivation {
  name = "test-derivation";

  buildInputs = [ pkgs.python36Packages.jupyterlab
                  pkgs.nodePackages."@jupyterlab/toc"
                  pkgs.nodejs];

  shellHook = ''
    export TOC_DIR=${pkgs.nodePackages."@jupyterlab/toc"}/lib/node_modules/\@jupyterlab/toc
    TEMPDIR=$(mktemp -d -p /tmp)
    mkdir -p $TEMPDIR
    cp -r ${pkgs.python36Packages.jupyterlab}/share/jupyter/lab/* $TEMPDIR
    chmod -R 755 $TEMPDIR
    echo "$TEMPDIR is the app directory"

    # labextensions
    jupyter labextension link --no-build --app-dir=$TEMPDIR $TOC_DIR 
    '';
}

[costrouc]

Good! So it looks like link will work just fine. Also if --no-build works this means that it could be very cheap (timewise) to build jupyter editors with different kernels and extensions

[MMesch]

Here is a working jupyterWith derivation:

MMesch@76ed1a5

you can run it with:

nix-build -E '(let pkgs=import ./nixpkgs {}; in pkgs.python36Packages.jupyterlabWith [ pkgs.nodePackages."@jupyterlab/toc" ])'
nix-shell -E '(let pkgs=import ./nixpkgs {}; in pkgs.python36Packages.jupyterlabWith [ pkgs.nodePackages."@jupyterlab/toc" ])' --run "jupyter lab --app-dir=$out"

Do you want to pull these commits in your PR @costrouc ?

I think the proof of concept is convincing. To make this useable, we need to have all the functionality that you have in your other PR. Also we should probably pin the versions of the extensions if possible.

[MMesch]

latest commit:

https://github.com/MMesch/nixpkgs/tree/MMesch/jupyter-npm-extensions

with this shell.nix:

let 
  pkgs = import ./nixpkgs {};
  jlab=pkgs.python36Packages.jupyterlabWith [ pkgs.nodePackages."@jupyterlab/toc" ];
in
pkgs.stdenv.mkDerivation {
    name="customJupyterlab";
    buildInputs=[ jlab ] ++ jlab.propagatedBuildInputs;
    shellHook="export JUPYTERLAB_DIR=${jlab}";
  }

it can be started with nix-shell --run 'jupyter lab --app-dir=$JUPYTERLAB_DIR"'. Do you think we should work on the same branch @costrouc ?

[costrouc]

Thanks for this work! It will need to be added to the Jupiter editor derivation https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/editors/jupyter. Most likely an extensions.nix will be added. Also some extensions are also python modules which will need to be figured out. I will have time this afternoon to look at that

[costrouc]

I have a semi working implementation of this. I got @jupyterlab/toc to add but some other extensions are not as trivial. I will be closing this issue to create a much cleaner one with it working properly. Should see a PR in about a week. I will CC you. I will also be closing this along with two other PRs to combine with this one. #49807, #49721, and #50858.

This will lead to a PR with better python 3 support for jupyter (newest releases are python 3.5+ only). Addition of 3 kernels (c, go, ansible) and 20 jupyterlab extensions. Also will have an editor that has a clean way to add kernels and extensions. This will be a big PR but I am finding that they all depend on each other.

[MMesch]

looking forward to seeing it!

[MMesch]

any news from this @costrouc ? Feel free to tell me if you could use help.

[costrouc]

I could use help. But it was harder than I initially thought. Using the link argument does not work well because it often requires some development npm package for building. I would need to understand the npm/labextension build process better before moving forward with this.

[MMesch]

Do you have a branch that I could look at for experiments?

[costrouc]

Not sure if it will be much help but here is the big PR that I was working on https://github.com/costrouc/nixpkgs/tree/costrouc/jupyterlab-extension-kernels

[alexvorobiev]

Does this recent work by TweagIO help? https://github.com/tweag/jupyterWith

[MMesch]

We understood the build process of jupyterlab quite well now I think. But it is not clear to me yet how this can be integrated into nixpkgs. The crucial point right now is still how to add the extensions:

When an extension is added to jupyterlab (e.g. with jupyter lab install jupyterlab-ihaskell) it runs the yarn resolver to determine compatible versions of all Jupyter core modules, all installed extensions and their dependencies. It then produces a yarn.lock file with these versions and downloads the extensions and dependencies from npmjs.org, github or the local disk as a tar package. A link to this local tar file is in the yarn.lock file. Afterwards webpack bundles and minifies the code in a static folder that is then executed in the browser.

It is the resolver step that is most difficult to reproduce with nix. Probably we won't be able to do so. For jupyterWith, we therefore jump in after the resolver and prebuild jupyterlab with the required extensions with its own build process. We also added an impure nix function wrapper around this but it can't be run when sandboxing is activated.

So how could we integrate this into nixpgs? I am not sure and I certainly don't have enough experience for it. The only idea that we had so far is to resolve jupyterlab with all extensions that we can find. The resolved yarn.lock file could be distributed with nixpkgs and built with yarn2nix (for example) and webpack. All extensions could be disabled by default and the user selects the ones that he wants. (We should explore jupyter labextension disable xxx)There are a few questions that remain open in this case. For example, what should we do with local extensions? Probably an option to add a custom prebuild jupyterlab would still be useful.

Also linking tweag/jupyenv#4 and @guaraqe .

[costrouc]

@MMesch thanks for posting here! I was looking at your work over the weekend and I REALLY appreciate it. Wish that I had more time at the moment to help with that specific project. Yes when I was working it on it in the past I too remember jupyter labextension install to be the step that is going to be really hard to deal with. If I understand it correctly you install a set of extensions say jupyterlab-git, jupyterlab-xkcd, jupyterlab-toc, jupyterlab-latex. The easy part is that several of these extensions require python packages installed. But the hard part is that jupyter will download the npm packages for each of these extensions and compile then with yarn into a static folder. This means that a unique yarn install build is needed for every subset of packages a user wants of the jupyterlab extension ecosystem. @MMesch does this state the hard part of the problem?

I have some connections with jupyterlab development and I would like to talk to the people there about this problem.

[saulshanabrook]

Hey all, I work on JupyterLab and wanted to chime in here. You both seem to understand the situation pretty well.

This means that a unique yarn install build is needed for every subset of packages a user wants of the jupyterlab extension ecosystem. @MMesch does this state the hard part of the problem?

Exactly. Every combination of package versions and JupyterLab versions requires a new build and they are not composable. What I mean is that you can't build a package with a certain version of jupyterlab and somehow combine that with building another package with that some version of jupyterlab. The only way to combine it is to install both extensions and run build.

We have many related issues open around this topic, but I think the most relevent is jupyterlab/jupyterlab#5672 "Create and install extensions without Webpack".

The TLDR is that when we install multiple extensions we de-duplicate and flatten the resulting dependency tree and then build that all together into one output. This means changing one extension requires rebuilding everything.

If anyone wants to put cycles towards fixing that in core, or wants to explore that space, I will happily help them get up to speed on that and work with them.

[MMesch]

Hi @saulshanabrook , good to have your opinion on this!

I just want to clarify that I see the resolver step as the problem and not the build process. The yarn resolver makes everything impure. Once we have a yarn.lock file with the precise dependencies, we can probably build with yarn2nix and webpack ourselves. On the other hand, you could argue that we wouldn't win anything compared to distributing a prebuilt app directory. The only advantage that I see is that we could include a yarn.lock into nixpkgs but hardly any prebuild javascript code. As a dirty hack, we could then potentially deselect extensions that we don't want with a nix function, which brings me to my second point:

could we resolve a large set of extensions and then just build with a few of them? Or deactivate them selectively from the web interface? This would allow us to make the whole derivation pure because it requires only a single yarn.lock instead of all version combinations. We would just have a single set of curated extensions that work together from which the user can chose the ones that he wants. Of course this doesn't work if someone wants to install a custom extension.

I think jupyterWith works quite well so far. Derivations are cached, we can compose kernels with packages and libraries and we can have extensions (with an impure step before caching). What is missing is how we can make this completely pure and how we can include it in nixpkgs without distributing prebundled javascript code.

On the other hand, I was wondering if it wouldn't be easiest to just distribute a Jupyterlab version with a list of curated extensions. A bit like inkscape, gimp, dia or other programs that are packaged for linux. This might also be interesting for other use cases. Would the Jupyterlab team be interested in this @saulshanabrook ?

[MMesch]

@costrouc : How do you think we should proceed? Should we start implementing other kernels (for example including your work) in jupyterWith to setup something that works for now? We can merge the whole package into nixpkgs once it is clear how to deal with the extensions. The advantage would be that it is usable immediately, and we can also experiment a bit more with the code before getting it into the main repo. Another option would be to immediately work on a jupyterWith function in a nixpkgs branch and work on it there.

[costrouc]

@MMesch you are right the non deterministic step is the yarn lock file. When I was working with fixing the version of each extension I ran into issues with npm vesions of each package being incompatible with the jupyter lab version. I think that lab extensions at this point are a hard problem to solve and would require a lot of work on nixpkgs side or work on jupyterlab's side. I don't think this is something we could hope to solve easily and we already know it has an easy fix of using an impure derivation which I don't think is a deal breaker.

So maybe focus on the areas that we know we can improve? Support for more kernels (making them easy to include) and adding the python packages that are included with some labextensions? For example jupyterlab-git requires a pypi package jupyterlab-git. This can be done taking advantage of passthru. See https://nixos.wiki/wiki/Workgroup:DataScience where we inspect packages if they require labextensions

[MMesch]

ok I agree. Let's fill up jupyterWith then. Shall we somehow integrate the work that you have done in #49807 ? In parallel we can setup a nixpkgs branch to test how it can be integrated smoothly ...

[costrouc]

Yes I think there are a few new kernels in that PR. If I remember right nixpkgs will require a few additional packages to have those kernels.

[saulshanabrook]

could we resolve a large set of extensions and then just build with a few of them? Or deactivate them selectively from the web interface? This would allow us to make the whole derivation pure because it requires only a single yarn.lock instead of all version combinations. We would just have a single set of curated extensions that work together from which the user can chose the ones that he wants. Of course this doesn't work if someone wants to install a custom extension.

This sounds like a great idea! I think creating a large JupyterLab distribution with a bunch of extensions installed would also be generally useful to the larger community.

One way to package this to make it accessible to folks outside of the nix ecosystem would be to publish PyPi and Conda packages that have all the other packages installed (i.e. ship with the built JS files). @jasongrout do you know of any other efforts creating a large JupyterLab distribution with a bunch of pre-included extensions?

We have a "Extension Manager" that you can enable to allow disabling/enabling (along with installing/uninstalling) extensions from the UI:

[jasongrout]

@jonmmease worked on a jlab distribution something with lots of plugins installed. There are probably also various institutional efforts that are doing similar things in their internal platforms.

[costrouc]

This is great thanks for sharing! Definitely something I plan to check out and see how this can be integrated with nix.