Examples as modifiable notebooks

[rubiop]

I think we found a nice procedure to produce python notebooks and build them in the documentation. To do that we use .py created via files Jupytext. Those .py files can be ran by anyone from the example folder but text is displayed as comments and we loose the cell interaction that we normally have in notebook.s I am curious to hear your thoughts but I think I will really like to have, in addition to the .py files the equivalent in .ipynb somewhere in git repo so we can link the modifiable notebooks in the documentation. Notebooks can easily be generated from .py using Jupytext but I think it will be better if this additional conversion step is not left to do by the potential new MParT/transport map user.

We also plan to have modifiable notebooks for Julia and Matlab so it might make sense to have the same in Python.

Let me know what you think about that.

If it make sense, there might be a way to automatically generate .ipynb files from .py files by github actions when modifications are made on these files during PRs.

[bonh]

I agree that notebooks are a nice way to combine interactive code with text. But using version control directly with notebook files can be annoying, see here for details. That's why I refrain from putting notebooks directly into a GitHub repo. I do not know how this issue works out with Julia or Matlab notebooks.

But as you said, it is easy to generate .ipynb from .py automatically using actions and it is already done here and here while building the binder or the docker (it just does not work at the moment due to MeasureTransport/MParT-examples#24). So if a user wants to download the notebook they can access it via the binder. And in the documentation, we can link to the binder.

[mparno]

I think it would be nice having the notebooks in the repo directly. I don't think version control will be too much of a headache if we don't update the examples much. Right now we're jumping through a lot of hoops and headaches to avoid version control of the notebooks (jupytext, pulling the rendered notebooks from binder, etc...) and I'm starting to think it might be less cumbersome to just have the notebooks in the repository. It would certainly make building the documentation easier (locally and in github actions) and would allow "rendered" notebooks to appear in MParT-examples on github.

[dannys4]

could we possibly store the ipynb files in the github and use a jupytext action to convert them to .py files, then use those for testing the examples in our workflow?

[rubiop]

I agree that notebooks are a nice way to combine interactive code with text. But using version control directly with notebook files can be annoying, see here for details. That's why I refrain from putting notebooks directly into a GitHub repo. I do not know how this issue works out with Julia or Matlab notebooks.

But as you said, it is easy to generate .ipynb from .py automatically using actions and it is already done here and here while building the binder or the docker (it just does not work at the moment due to MeasureTransport/MParT-examples#24). So if a user wants to download the notebook they can access it via the binder. And in the documentation, we can link to the binder.

Ok I see, for me we won't have to do version control on the .ipynb files since they will be automatically modified via github actions when we modify the .py files that are version controlled.

Link to the binder in the documentation will be nice indeed.

[dannys4]

I, too, cannot speak to matlab live scripts, but just FYI @rubiop, a pluto notebook is just a .jl file which you could easily run on the command line with julia (whereas .ipynb files cannot be run with python), so it's a bit of a different situation regarding version control.

[mparno]

If running the notebook for testing is the issue, we can just use papermill. That's what we do for MUQ.

I might be wrong, but I think the original motivation for using *.py files and jupytext was more about being able to interpret the diffs produced by git. Because of all the extra information in a notebook, a git diff of the notebooks is harder to interpret.

[rubiop]

For Matlab the framework is exactly the same as python. In one hand you have .mlx files which is live script (=notebook) format. This format is not understood by git but works well with the built in version tool in Matlab. In the other hand you have .m files that git understand but where text and equations are not formatted. It is very easy to go from one to an other.

For example you can create your live script in the .mlx format, save it as a .m file and all the text (with formatting commands) will appear as Matlab comments. Then, you can convert the .m file to the original .mlx file just by "opening it as live script" in Matlab.

Long story short: we can do the same thing than in python and we will probably want to use the same workflow type

PS: for now I am will only work if .m files in the git repo and we will decide if we want the corresponding .mlx files instead or in addition.

EDIT: during the double conversion .mlx->.m->.mlx there is some text formatting loss: title and heading 1 are transformed into heading 1. Headings 2 and 3 are transformed into normal text

[dannys4]

Not sure this is the right discussion, but I just wanted to publicly share a perspective on Jupytext-- both @rubiop and I have to translate examples from python to our respective bindings, and the jupytext-formatted scripts make it extremely nice for translation purposes, to the point that we both have written/are writing scripts to do that job for us. As we add more examples, we won't have to expend nearly as much effort translating to these languages if we have prebuilt scripts to translate (most) of the notebook for us.