Swift-Jupyter

This is a fork of google/swift-jupyter. It is made possible to use Jupyterlab (as well as Jupyter Notebook) with most up-to-date Swift toolchain.

This fork made several significant changes to original repository organization. It biased towards using Bazel as code and dependency management tool. This fork is fully separated from Swift for TensorFlow project and is actively maintained to keep everything working with up-to-date Swift toolchain.

Right now, this fork worked well with Swift from 5.4.x to 5.5.x on Linux without any tweaks (if installed to /opt/swift). It is tested with Ubuntu 20.04 distribution, but should work with other flavors.

Because both the notebook integration and PythonKit are used actively with one of my side business, the continuing maintenance of this repository can be assured.

I do want to make this work on macOS. It should be only a few OS / toolchain detection script away.

Getting Started

Requirements

Operating system:

• Ubuntu 20.04 (64-bit); OR
• Other operating systems may work, but you will have to build Swift from sources.

Dependencies:

• Bazel
• Python 3

Installation

Extract the Swift toolchain to /opt/swift.

Inside the swift-jupyter repository:

./scripts/bazel/setup_clang.sh /usr/local

Note: /usr/local should be where your bin/llvm-config binary resides.

Now you can run Jupyterlab:

SWIFTPATH=/opt/swift bazel run //lab:bootup -- --notebook-dir $PWD

To check if installation is done, run the notebooks/_swift_template.ipynb notebook.

Usage Instructions

Rich Output

This fork removed EnableIPythonDisplay.swift. Using IPython kernel and its interactive shell as in google/swift-jupyter cause some silent crashes with the most up-to-date Swift toolchain. It only happens after many tens of cell executions.

Instead, this fork enhanced EnableJupyterDisplay.swift to enable rich output from Python (e.g. Pandas or matplotlib). Because communication with Jupyter notebook requires HMAC computation, this fork updated EnableJupyterDisplay.swift to use apple/swift-crypto for these cryptography primitives.

The EnableJupyterDisplay.swift now will be automatically included (enabled) once your package installation is done.

You can call Python libraries using PythonKit. I've set up Bazel with both PythonKit and swift-crypto support.

To see these in action, first install these packages with Bazel inside your notebook:

%bazel "@PythonKit//:PythonKit"
%bazel "@SwiftCrypto//:Crypto"

Now you should be able to display rich output! For example:

let np = Python.import("numpy")
let plt = Python.import("matplotlib.pyplot")

let time = np.arange(0, 10, 0.01)
let amplitude = np.exp(-0.1 * time)
let position = amplitude * np.sin(3 * time)

plt.figure(figsize: [15, 10])

plt.plot(time, position)
plt.plot(time, amplitude)
plt.plot(time, -amplitude)

plt.xlabel("time (s)")
plt.ylabel("position (m)")
plt.title("Oscillations")

plt.show()

let pd = Python.import("pandas")
pd.DataFrame.from_records([["col 1": 3, "col 2": 5], ["col 1": 8, "col 2": 2]]).display()

Build Swift Libraries that Support Jupyter Notebook

swift-jupyter now provides a package called JupyterDisplay that you can included in your library to support rich outputs inside Jupyter Notebook. I've been successfully implemented nontrivial interactive streaming GUI with this capability.

If your package has dependency to JupyterDisplay, when used inside swift-jupyter, your query to JupyterDisplay.isEnabled will be true and you can use JupyterDisplay.display functions to send HTML, PNG images or plain text to Jupyter Notebook. These can be flushed periodically with JupyterDisplay.flush() method.

Code Completion

This fork supports simple code completion with sourcekit-lsp now shipped along Swift toolchain. The code where uses Swift for TensorFlow's LLDB for code completion is not removed if you choose. I may remove that support entirely once the sourcekit-lsp challenge is resolved:

Current sourcekit-lsp integration relies on appending to a hypothetical file on each successful cell execution. However, REPL execution is a bit different from code in a file because we can shadow variables without causing any problems. This can eventually leads to a hypothetical file that AST is not well-formed. For these cases, sourcekit-lsp based code completion will be degraded into a token-based code completion engine.

Jupyterlab Integration

If you manage your repository with Bazel, integrate with swift-jupyter provided Jupyterlab is the best way to run Jupyter Notebook. It keeps Python dependencies clean, can reference to Python packages either through pip or in-tree Python packages through Bazel. Once it sets up, your repository Jupyterlab configuration is portable, you don't need to worry about installed extensions not available in a different computer.

First, add swift-jupyter to your WORKSPACE such as:

git_repository(
    name = "swift-jupyter",
    commit = "daf4eef0ea20be0d6eec5306b5b1cfdb11550d1e",
    remote = "https://github.com/liuliu/swift-jupyter.git",
    shallow_since = "1658788905 -0400",
)

# You do need to include Python support package for Bazel, if you haven't already:

http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.4.0/rules_python-0.4.0.tar.gz",
    sha256 = "954aa89b491be4a083304a2cb838019c8b8c3720a7abb9c4cb81ac7a24230cea",
)

load("@rules_python//python:pip.bzl", "pip_install")

# Install Python dependencies

pip_install(
    requirements = ":requirements.txt",
)

Second, you need to modify / add requirements.txt file under ./external/ that includes Jupyterlab reference.

Third, just find a place to use the provided jupyterlab macro to create a target, for example, this is what we do inside ./lab/BUILD:

load("@swift-jupyter//:lab.bzl", "jupyterlab")

jupyterlab(
    name = "bootup",
    deps = [],
)

You can add more dependencies through deps parameter. To launch Jupyterlab, simply do:

SWIFTPATH=/opt/swift bazel run lab:bootup -- --notebook-dir $PWD

Troubleshooting

This fork uses Bazel to setup build and run environment. It helps to maintain a consistent environment between machines. There are two varying factors could impact the environment on Linux:

1. clang

2. Swift toolchain

Current script assumes clang installed under /usr/local/bin, which should be a reasonable assumption but can vary. If you have LLVM toolchain installed somewhere else, ./scripts/bazel/setup_clang.sh run should take a different prefix other than /usr/local.

Bazel doesn't inherent environment variables. Thus, your Swift toolchain location needs to be passed in through command-line every time. Make sure it is the right location for your Swift toolchain. On Linux, that path should contain a sub-directory called usr.

Compile Swift LLDB with Python Support

Official packages provided through https://www.swift.org/download/ can be a hit-or-miss. There are some CI issues that I actively work with Swift members to make sure Swift LLDB is compiled with Python support. However, they can be missed from time to time. Here is a documented instruction for how to compile Swift LLDB with Python Support from source:

1. Make sure you have enough disk space, and an empty directory for source code from various repositories, I normally use /opt/swift;

2. To make sure you will build with Python support, you need to install Python's header file. On Ubuntu, it is apt install python3-dev;

3. git clone the following repositories to that directory:

   • https://github.com/apple/swift

   • https://github.com/apple/llvm-project

   • https://github.com/apple/swift-llbuild (Please rename to llbuild)

   • https://github.com/apple/swift-cmark (Please rename to cmark)

   • https://github.com/apple/swift-driver

   • https://github.com/apple/swift-tools-support-core

   • https://github.com/apple/swift-corelibs-libdispatch

   • https://github.com/apple/swift-argument-parser

   • https://github.com/jpsim/Yams (Please rename to yams)

4. Make sure these repositories are checked out with the matching branch. For example, if you want to compile for Swift 5.6.x, it should be branch release/5.6. For llvm-project, it is swift/release/5.6. It is OK to be on the tip for Yams and swift-argument-parser;

5. Create an empty build directory at the same level as all other repositories, such as /opt/swift/build;

6. Go to swift/utils, and run ./build-script --release --lldb;

7. You should be able to find the relevant LLDB files under /opt/swift/build/Ninja-ReleaseAssert/lldb-linux-x86_64.