Venus (Jupyter but lighter, faster, and hotter)

[skyepurchase]

Your name, department, and University

Skye Purchase, Mathematics, Imperial

Name(s) and department(s) of anyone else relevant to this project

No response

Please write a brief description of the application area of project

Improved collaborative programming, version control and HPC notebooks.

Notebooks, based on the Jupyter notebook framework, are prevalent across scientific computing with many benefits including interleaved markdown explanations/descriptions along with python (or a limited range of other languages) code. They allow for experimentation, visualisation whilst still being a "runnable" program.

However, these do not do well on HPC due to their reliance on JSON file storage causing huge problems with git versioning and collaborative coding. Furthermore, the result .ipynb file is NOT a program file and cannot be run outside of a lab or notebook environment.

Please describe the project.

This is a long term project to design an alternative to Jupyter notebooks that is programming file native (that is the resulting file will be a C, Python, Rust, etc. file rather than a .ipynb) but has the benefits of Jupyter notebook visualisation and inline explanation.

The basic idea is to have a standard that allows for markdown and program code to be interleaved without relying on JSON objects. The resulting files can be read as markdown files (depending on program images may or may not appear, think terminal vs IDE) or executed as bonafide program files.

It would be good to have interoperability with Jupyter (people will not use it if they have to rewrite all their projects again) as well as something akin to Jupyter Lab for improved visualisation and coding. In the long term it is likely that having some kernel (like the IPython kernel) will be useful.

What will be the outputs of this project?

In the long term a full standard that can be used across vendors that is based on markdown and program files rather than an intermediate JSON file. This will include a standard for writing .md files that can be converted to native program files (easily by hand or automatically by a program) or native program files with commented markdown elements; as well as an IDE that visualises the code and markdown cells similar to Jupyter.

The project will include a method to convert Jupyter files into these new file standards and vice versa (though going backwards isn't a major priority). This program will also be able to convert the files into PDFs, markdown, or native program language files, as well as run the file as a program.

A language server protocol for Venus so that VSCode, Vim/Neovim, Jet Brains, etc. can write Venus files with code completion hints and shortcuts.

Which programming language(s) will this project use?

C, Lua, Typescript, Vimscript (to understand the existing implementation)

Links to any relevant existing software repositories, etc.

The inspiration for this project: vim-venus but it's written in vim-script and only works in vim/neovim. This will be the starting point into a more general standard.

Markdown formats for jupyter include jupytext but this is only a python library as well as neovim plugins jupynium which uses jupytext and the ipython kernel.
I'm sure there are more, but all of them still try to be jupyter notebooks just in a markdown format rather than fully fledged markdown approaches.

Links to any relevant papers, blog posts, etc.

I haven't read much around the subject so haven't found any blogs, there are definitely no papers on this.

Make project public

• I understand that this project proposal will be public

[skyepurchase]

I haven't worked out exactly what the next steps will be.

I think MVP is getting a precursor standard that we can then convert between markdown and python files. Maybe extend this to other languages.

After this look at converting between Jupyter and this standard, before moving into creating our own language kernels, LSP, and web based IDE. This will likely be future projects across the next few programming weeks across cohorts.

[skyepurchase]

I am considering moving the development to Rust (because I want to learn it). Also the Javascript aspect will be much later when developing a similar environment to jupyterlab.