Epic Roadmap to new version of Dnotebook

[risenW]

We decided to rewrite Dnotebook to be more like Jupyter notebooks, where we can write and run code that interacts directly with the user's local OS in a safe and secure way. This overarching goal here is that we want users to have a native experience when using Dnotebook.

The following is a list of proposed features for the new release:
Target release date: March 2022

• Rewrite core components in Typescript
• Add a new code editor with syntax highlighting, lining, snippet, and autocomplete.
• Add support for file system directory: Users can interact with their local file system from the UI so they can create, update and delete files securely
• Bash command support for managing notebooks: Users can execute bash commands from the UI. This makes the installation of new packages relatively easy and intuitive.
• Better UI/UX
• File menu options for managing kernels and notebooks

[risenW]

Some musings on design architecture so far:

Update on the server side

The first week, I started by asking the question, how do I make Dnotebook work like Jupyter, meaning bash command works so I can run yarn add package in the browser, execute it server-side in Nodejs, and get real-time output logs in the browser.

I decided to go with the express HTTP stream. Nodejs has great support for streams, so I would basically start a server, send commands via HTTP request, open a stream response, capture logs from stdout and stderr, and continuously write back a response to the UI. This is basically done at the moment. see https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/routes/code_routes.ts

Now for the next task, I wanted to execute JS code. Executing JS code one time and returning the result is pretty straightforward, we can use an eval for that. But what I needed is a way to execute each cell in the same context. I.e All cells from the notebook will share the same variables-the typical notebook behavior So what do I do here?

Well, interestingly Nodejs has the VM API, which you can use to compile and execute code in a given context. So basically what I did was start a server, initialize a context, then send chunks of code to be executed in that context. Because the server keeps running between execution, the context is kept alive! This was implemented here: https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/runner/nodejs.ts

So with the core functionality done, I decided to add some more features.

First, I added an inline transpiler, meaning you can write any flavor of JS, and mid-way before it gets executed, it is transpiled using Babel to a compatible version that can run on the VM. See implementation here:

dnotebook/src/server/src/runner/nodejs.ts

Line 39 in f8a65fd

let transformedJsCode = transformSync(code, {

So what next? Well, a lot of things still has to be done on the server-side, some of the important issues I’m yet to figure out is:

1. How to track the state of cells and know when they finish execution? This is quite tricky given the async nature of JS

2. Managing multiple child processes-Kernels in Jupyter notebooks-so I can easily start and stop them from the UI

3. Security of code process

Update on the UI

At the moment, it supports full editing of Markdown, JS, HTML, bash, and JSON. That means, syntax highlighting, code snippets, autocomplete, and execution all work at the moment. I used AceEditor for this. See implementation https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/frontend/components/CellEditor/index.tsx

So what’s next here? Well, the next major task we want to tackle in the UI is:

• Having a local file directory displayed in the browser, so users can create and manage files directly from the browser. @AceTheCreator has decided to work on this. See the new issue here: Add file system tree support  #11

I will keep updating this comment as we add new features and encounter weird issues. This will be used to create a detailed architecture document before release.