What options exist for extending markdown cells to support reports on multiple kernels? [Was: Can customized kernel hijack markdown cells?]

[BoPeng]

I am working on a kernel where the underlying workflow system can generate reports based on results of other steps. For example,

! ## samples
! There are ${nsamples} samples with names ${sample_names}

would generate

## samples
There are 2 samples with names sampleA, sampleB

I would like to store these report lines in jupyter as markdown cells but currently the kernel can only handle script cells through do_execute. Is there anyway that I can hijack markdown cell, change its content (expand ${samples} etc) before it is rendered?

[Carreau]

I would like to store these report lines in jupyter as markdown cells but currently the kernel can only handle script cells through do_execute. Is there anyway that I can hijack markdown cell, change its content (expand ${samples} etc) before it is rendered?

No, not currently. We had the discussion a long time ago, and want something similar and generic across languages. The current conclusion is that we will not allow it before markdown/common mark have a syntax that allows extensions, otherwise you end up writing your own parser, what we don't have the bandwith to do.

You can (of course) decide to monkey patch the notebook with the custom extension, we can't prevent you to do that.

[BoPeng]

I went through the discussions and I have to agree that jupyter should really try to resolve this issue to make the reports more dynamic, and thus more useful. I disagree with your point that jupyter has to provide a language-neutral solution because variable and expressions from different languages take different forms and it is impossible to settle down with a common syntax. All jupyter needs to do is provide a do_markdown api to give each kernel a chance to handle markdown cells differently. In my case, I would simply do

def do_markdown(self, cell):
    cell.source = interpolate(cell.source, self.my_dict)
    return super(SoS_Kernel, self).do_markdown(cell)

In the case of python, with the introduction of new format string to version 3.6, there is no reason to think of any other syntax.

[takluyver]

Hiding code cells is also a possibility, which would make it easier to do such things as the output of a hidden code cell.

[jasongrout]

There is a hack to do this: If you use a PlaceProxy widget, you can replace any html id tag with a widget. So you could in your markdown cell do:

## samples
There are <span style="display:inline-block;" id="nsamples"></span> samples with names <span style="display:inline-block;" id="sample_names"></span>

and then in your code cell, do

from ipywidgets import PlaceProxy, Label
from IPython.display import display

nsamples = Label('5')
sample_names = Label("'name1'")
display(PlaceProxy(nsamples, selector="#nsamples"))
display(PlaceProxy(sample_names, selector="#sample_names"))

Then if you set, for example, nsamples.value='10', that should update the markdown.

This is really fragile - it depends on the markdown being rendered to html on the page when the widget is displayed. It doesn't persist to the notebook file. But it is a kludgy way to do what you're asking.

[jasongrout]

@BoPeng - markdown cells are never sent to the kernel in the current notebook. They are rendered completely in the browser.

[BoPeng]

I understand it now, but this does not have to mean it cannot be changed. I share the opinion with many others (ipython/ipython#2592) that RStudio is miles ahead of Jupyter for generating pretty reports and this is one of the key features that are lacking. The issue was discussed back in 2012 and I do not understand why it was never taken seriously (apologize if I am wrong here).

[BoPeng]

@takluyver Yes, that would more or less work, but it is not ready, right? (I can see jupyter/notebook#534). I do not know how that operates, but users can enter

! ## samples
! There are ${nsamples} samples with names ${sample_names}

as code, evaluate it, and hide the cell. It is not as convenient as entering

## samples
There are ${nsamples} samples with names ${sample_names}

as markdown though.

[takluyver]

It's not that we didn't take it seriously, it's that it's not obvious how to make it work intuitively, and there are thousands of things we want to do.

Apart from the syntax issue Matthias mentioned (we really don't want to go making more ad-hoc special symbols for Markdown), there's an interface question - do we expand the variables in a Markdown cell when we render it, or keep them up to date as code is executed? The former is technically much simpler, but Markdown cells, unlike code cells, have no visible code or prompt number to indicate that they might need to be re-executed. Should they disappear or stay when the kernel is restarted, the page is reloaded, or the notebook is converted to another format? A format like knitr has this much easier, because it has a clear source document and output document. If that's what you want, have a look at knitpy, or the cross-language tool dexy. Doing this interactively is more challenging)

(I'm sure you can come up with answers for the questions above, but my point is that the answers aren't obvious - something has to be hashed out between different use cases and priorities)

[BoPeng]

we really don't want to go making more ad-hoc special symbols for Markdown

I read arguments from both sides but I am on the side that we are NOT creating another special symbol for markdown. Because each language has its own syntax and there is no way we can find a commonly accepted one, the decision on a syntax should be made at the kernel level. The kernels decide what special symbols/syntax to use and how to translate them to proper markdown syntax . I mean, Jupyter does not have to add or understand anything. It simply feed the markdown cell to the kernel (if do_markdown is defined) and render the returned markdown text.

there's an interface question - do we expand the variables in a Markdown cell when we render it, or keep them up to date as code is executed? The former is technically much simpler, but Markdown cells, unlike code cells, have no visible code or prompt number to indicate that they might need to be re-executed.

No. The markdown cells are evaluated with shift-enter and users can re-evaluated them if needed. The users could run-all-cells to generate a final updated document.

Should they disappear or stay when the kernel is restarted, the page is reloaded, or the notebook is converted to another format?

The markdown cells would be hide-source cells with both versions saved (like code cells), but only the evaluated version is displayed. This is exactly what users are seeing now: the markdown code is hidden unless we enter the cell.

A format like knitr has this much easier, because it has a clear source document and output document. If that's what you want, have a look at knitpy, or the cross-language tool dexy. Doing this interactively is more challenging)

I failed to understand why it is so challenging if we accept a bit of inconvenience (e.g. manual re-evaluation of markdown cells) caused by the interactive nature of workflow.

[BoPeng]

BTW, I understand @Carreau's argument that we are adding special syntax if we introduce special syntax to markdown cells of the python kernel, which can in theory cause existing documents fail to render. However, with the introduction of new hook (do_markdown), it costs almost nothing to introduce derived kernels such as

class MdPythonKernel(PythonKernel):
    def do_markdown(self, cell):
          return process(cell.source)

and let users decide which kernel they want to use. Again, it does not have to be jupyter's job to decide on special syntax for any kernel.

[jasongrout]

@BoPeng - what about doing this as a cell magic?

%%interactivemd

## samples
There are ${nsamples} samples with names ${sample_names}

Cell magics are sent to the backend where the markdown can be rendered to html and sent back to the browser. Then the only piece missing is easily hiding the input and easily showing the input if you want to edit the source.

[BoPeng]

Yes, this would certainly work. I just need to decide if I should add another way to specify report because users can already enter ! lines in code cells, and %report just adds another special code cell.

[Carreau]

Because each language has its own syntax and there is no way we can find a commonly accepted one,

That untrue, we want to insert value in markdown, so it's enough to find a caracter which work in markdown.

the decision on a syntax should be made at the kernel level

That an opinion, not a truth, and no after long reflexion we think that doing that at kernel level is a bad idea for many reason:

• it's blocking,
• it would needs the code to be reavaluated by nbconvert
• it requires each kernel to implement a markdown parser and renderer,
• Initial prototypes of the notebook were rendering markup cell server side, it was a nightmare.

I mean, Jupyter does not have to add or understand anything. It simply feed the markdown cell to the kernel (if do_markdown is defined) and render the returned markdown text.

That would require an enormous change in protocol, and break the main abstraction that kernels are only for computation, and there cannot be "if do_markdown" is defined, as this is a protocol you should conform to, and multiple client can be connected.
Plus as far as we know there are alternative frontends that do not use markdown at the markup language. You cannot for that on them.

No. The markdown cells are evaluated with shift-enter and users can re-evaluated them if needed. The users could run-all-cells to generate a final updated document.

Again that's your opinion, some people do not like this solution,
plus this would have some consequence in the front-end like blocking the rendering of MD cells,
which is basically a no-no.

I failed to understand why it is so challenging if we accept a bit of inconvenience (e.g. manual re-evaluation of markdown cells) caused by the interactive nature of workflow.

The concept is interesting, and does make sens in some use case, and it is perfectly reasonable in some workflow. We did went through some of your ideas a few years ago, and after reflexions and prototypes we really things these are not the solution we like. Plus we have to maintain backward compatibility.

So I think you underestimate the cost, advantages/drawbacks of such solutions, there are likely some pros and cons, and good ideas. Like I personally, of opinion that Shift-Entrer requerying variable to the kernel for a markdown cell is a good idea, with another key to render without changing the current value is one thing I like. But I recognize that for user it is highly confusing, and for non modal frontend, it would be a nightmare to come up with the right semantics.

So we would be happy to see prototype, but it's likely not a direction we will take, at least not soon.

Would be happy to proven wrong if you want to hack a prototype.

[jasongrout]

Yes, this would certainly work. I just need to decide if I should add another way to specify report because users can already enter ! lines in code cells, and %report just adds another special code cell.

I'm not sure what you mean here. What do ! lines have to do with an %%interactivemd cell, and what is %report?

[BoPeng]

I'm not sure what you mean here. What do ! lines have to do with an %%interactivemd cell, and what is %report?

Sorry for the confusion. In SoS, reports are marked by lines with !, which can be mixed with other statements. I had hoped to separate them as markdown cells but now I need to process them in code cells, in which they already work in the ! form. Using %%interactivemd (or another name %report that I would prefer) just assumes all content in the code cell should be prefixed by !.

As a side question, there is no difference between line and cell magics in jupyter notebook, right? Because notebook would not evaluate a line starts with %magic directly with enter, I just used % for all sos magics.