Question: best authoring workflow at the moment?

[dsyme]

I want to author a set of sample notebooks. What's the best authoring workflow at the moment?

1. What I really want is "open notebook, edit, save" in a usual document way

2. Assume the notebooks end up in a github repo. I don't mind if I edit them online or on my local machine. If editing online the results would have to be committed directly to github on save. If editing on my local machine I'd manually push to github.

3. I'm not collaborating with anyone, just authoring on my own

4. I'd really prefer not to "edit, download, copy, save" like I have to do in mybinder today.

The best paths today appear to be a choice between

1. "edit in my binder, download, copy to my local repo, push back to github"

2. "edit locally using a local install of .NET Interactive Jupyter"

There's also a recent Visual Studio Code extension which looks really interesting. However that doesn't allow me to edit and save .ipynb, just this new .dib document format. My documents are .ipynd so I'd have to add manual document conversions to my workflow. If the VSCode extension allowed me to edit, run and save .ipynb using .NET Interactive tooling that would be great.

[zyzhu]

I'll share my two cents.

I find Azure Notebooks has a good user story. There are many sample notebooks. You can easily fork one to your own profile and decide to share it or keep it private. It's much easier to start to run an idea than on binder. Very little boilerplate to worry about.

If dotnet interactive can be integrated there and Microsoft has conviction to spend money to improve it and market it, it can attract lots of users in data science and AI community. It might have better proposition than Google's colab as Azure notebooks already support R, Python and F# out of box. Colab's focus is still on Python. Maybe Microsoft has other products in their mind that I am not aware of.

But my observation is that nobody's taking care of this wonderful preview product by seeing piles of unanswered issue lists.
https://github.com/microsoft/AzureNotebooks/issues

[jonsequitur]

To provide some context on the .dib design so far I've opened #467.

This explains a few of the reasons we didn't support .ipynb directly, but we do support it currently via convert and export in our VS Code extension, and these will be available as command line operations as well:

We're also adding support for flexible outputs during execution, so the output of the automated running of the script could be, for example, markdown or HTML, with or without code inline.

Looking ahead when these capabilities are in place, we hope achieve these workflows:

• Edit .ipynb locally in Jupyter, seeing output in the notebook.
• Edit .dib locally in VS Code, seeing output in the notebook.
• Convert .ipynb -> .dib
• Convert .dib -> .ipynb
• Edit .dib online using a simple text editor.
• Run .dib from the command line, whether locally or on a CI server, then
  • exporting a Markdown or HTML file and linked resources (including images and JavaScript) that can easily be published online in GitHub or a static site.

[zyzhu]

Pile on my previous comment about Azure Notebooks. The following is the main objective from Google CoLab on its landing page.
https://colab.research.google.com/

What is Colaboratory?
Colaboratory, or "Colab" for short, allows you to write and execute Python in your browser, with

Zero configuration required
Free access to GPUs
Easy sharing
Whether you're a student, a data scientist or an AI researcher, Colab can make your work easier. Watch Introduction to Colab to learn more, or just get started below!

Can Microsoft at least aim at beating authoring experience better than that?

[brettfo]

I'll also add that we don't default associate with the .ipynb file extension because that could steal the file association from the Python VS Code extension. The notebook APIs are still in preview, so eventually we may be able to associate with .ipynb, provided VS Code gives us the ability to select the extension used to open (or reopen) a notebook, but currently that's not there; it's an all-or-nothing file association and the convert to/from commands were the best option we had to support Jupyter notebooks without forcing them all to be opened with our extension.

[dsyme]

@zyzhu This isn't the right point to discuss the whole Azure Notebook hosted colab service thing I think - that has a whole set of issues of its own. I'm asking for workflows for authoring using the existing tools or short-term variations on them.

Looking ahead when these capabilities are in place, we hope achieve these workflows...

So the problem is that none of those workflows really work for me. And it's really conceptually jarring to have to convert anything to anything - I just want to edit my *.ipynb and save them. It feels like it's the #1 workflow we need to support.

Why can't I just edit a .ipynb locally in VS Code? I don't mind if I have to set up the file association manually somehow

I could use jupyter but TBH local installations of Jupyter just feel so clunky to me when VS Code is sitting around as a potential editor.

The VSCode previews look really great, but if the basic workflow isn't simple it's not easy to use them in practice...

p.s. Re .dib - please, no new formats in this context :) I need to create, edit and share ipynb. Any other format is ephemeral and only confuses matters.... (I can understand the potential need for a new format in other contexts, but not this one).

[dsyme]

Basically I find any workflow that isn't as simple as

- open 
- edit
- execute
- ctrl+s 
- git commit -a && git push

really jarring. The inner loop of edit+execute+save has to be really tight, or else I'd favour using an F# script and F# interactive.

[jonsequitur]

Agreed that this is a simpler experience, and I think should be available with either file format. This is the current experience for .dib and could be achieved pretty trivially for .ipynb with a feature in the VS Code extension that continually writes to the .ipynb when the .dib is saved or when a cell is run (to capture outputs).

It's not the default for a couple of reasons:

• The .NET Interactive experience in VS Code notebooks doesn't require Jupyter or Python, and
• the .ipynb format isn't very friendly for diffs, online editing (e.g. directly in GitHub), or automation.

[dsyme]

This ... could be achieved pretty trivially for .ipynb with a feature in the VS Code extension ...

Great, we should enable this then. What's involved to make it happen? Would we need the .dib at all or could it be ephemeral? I'd prefer not to see it at all in this context.

I'd be happy to help make the edit-over-ipynb experience happen, just point me to the right places.

The .NET Interactive experience in VS Code notebooks doesn't require Jupyter or Python...

Would the auto-convert-and-save-and-load-ipynb experience require Jupyter or Python? I'd love a Python-free experience here. We should be able to read/write ipynb directly, right?

the .ipynb format isn't very friendly for diffs, online editing (e.g. directly in GitHub), or automation.

I understand the concern here. However I didn't really want to discuss the pros and cons of a new file format, since my question is definitely about editing ipynb....

Specifically my content is long-lived eductional content going into a collaborative data-science/AI/ML programming repo (DiffSharp) where other contributors expect and trust .ipynb. And, importantly, they will instinctively distrust any other file format - the presence of any unfamiliar file format will be a point of rejection for them, and they simply won't work collaboratively with .dib content, at least not yet.

So no matter what the pros and cons in this context I'm constrained to established file formats as the "golden truth" for the content. I simply have no choice in that.

p.s. The other workflow I have in mind is simply to edit FSX files with markup and use a manually written converter to convert to ipynb. We have a converter for this here https://github.com/fsprojects/fsharp-ai-tools/blob/master/tools/fsx2nb.fsx. (For F# data science/ML/AI projects then .fsx counts as an established file format.)

[jonsequitur]

Would the auto-convert-and-save-and-load-ipynb experience require Jupyter or Python? I'd love a Python-free experience here. We should be able to read/write ipynb directly, right?

Absolutely. We intend to work seamlessly with Jupyter without requiring it. Saving to and from .ipynb works without Jupyter right now.

It's not a goal to deter people from using .ipynb or push an alternative for Jupyter users, but just to reduce the complexity for .NET notebook users who don't want to install Jupyter.

@brettfo's point above is important:

The notebook APIs are still in preview, so eventually we may be able to associate with .ipynb, provided VS Code gives us the ability to select the extension used to open (or reopen) a notebook, but currently that's not there; it's an all-or-nothing file association and the convert to/from commands were the best option we had to support Jupyter notebooks without forcing them all to be opened with our extension.

We're working with the VS Code and VS Code Python teams to make these experiences complementary and this thread is great feedback on the experience we'd like to get to.

[dsyme]

Closing this out for now