Add #!metadata kernel

[brettfo]

Current state of the world

The addition of a metadata kernel will allow .dib notebooks to have cell specifiers separate from the well-known ones of #!csharp, #!fsharp, etc. E.g., if one cell of a notebook connects to a SQL database with:

#!connect mssql --kernel-name adventureworks "<my connection string>"

then the user adds their queries, the current resultant .dib will look like this:

#!csharp

#!connect mssql --kernel-name adventureworks "<my connection string>"

#!csharp

#!sql-adventureworks
SELECT * FROM CUSTOMERS

This gives an odd view to the user because the query cell isn't really C# (or SQL, or F#, ...), and the user specified that they wanted the new kernel to be named adventureworks, not sql-adventureworks. The ideal would be something like this:

#!csharp

#!connect mssql --kernel-name adventureworks "<my connection string>"

#!adventureworks
SELECT * FROM CUSTOMERS

however, it's not known at parse time that #!adventureworks is a kernel selector, so when loaded, the notebook would have a single cell that looked like this:

#!connect mssql --kernel-name adventureworks "<my connection string>"

#!adventureworks
SELECT * FROM CUSTOMERS

Proposal

Add a kernel named #!metadata which at runtime is a no-op, but at parse time can specify various bits of document metadata, possibly expected to be valid JSON. The example above would result in something like this:

#!metadata
{
  "kernels": [
    {
      "name": "csharp",
      "language": "csharp"
    },
    {
      "name": "fsharp",
      "language": "fsharp"
    },
    {
      "name": "adventureworks",
      "language": "sql"
    },
  ]
}

#!csharp

#!connect mssql --kernel-name adventureworks "<my connection string>"

#!adventureworks
SELECT * FROM CUSTOMERS

The parser knows about the special #!metadata kernel, parses its contents, and it suddently knows that further down the file that #!adventureworks is a cell splitter and its language is SQL.

[mikoskinen]

Is it possible to use this kernel to add any custom metadata to .dib notebooks? In the examples metadata section is used to define kernel info and the #2217 mentions environment variables. We are currently doing some parsing to find the run arguments for a notebook and it would be create if these could be added to the metadata section.

In some cases it would be nice to add information like "description" and "tags" to the .dib file too. So if possible, it would be great if we could add anything we like to the metadata section.

[jonsequitur]

Yes, it will enable arbitrary metadata and make it available via the InteractiveDocument.Metadata property as (currently) a Dictionary<string, object> where the objects will typically be JsonElement. I haven't found a compelling reason yet to make this an actual kernel, though providing a way to get to the information at runtime within the kernel might be useful. We'd welcome input on the design.

We are currently doing some parsing to find the run arguments for a notebook and it would be create if these could be added to the metadata section.

I'd love to hear more about this scenario and how you plan to approach it. We're planning on enabling @input:-prefixed magic command tokens to be supplied via the command line during notebook automation.

Related: #1554, #2118

[mikoskinen]

Yes, it will enable arbitrary metadata and make it available via the InteractiveDocument.Metadata property as (currently) a Dictionary<string, object> where the objects will typically be JsonElement. I haven't found a compelling reason yet to make this an actual kernel, though providing a way to get to the information at runtime within the kernel might be useful. We'd welcome input on the design.

Thank you for the reply and for the details, this sounds good.

We are currently doing some parsing to find the run arguments for a notebook and it would be create if these could be added to the metadata section.

I'd love to hear more about this scenario and how you plan to approach it. We're planning on enabling @input:-prefixed magic command tokens to be supplied via the command line during notebook automation.

Our current solution only works with C#. We are using Roslyn to analyze the code from the first code cell of the DIB-file and then using a convention where a declared property = run argument. Roslyn is then again used to remove the properties from the cell.

Here's an example notebook:

#!csharp

int Age { get; set; }
string Name { get;set; }

var result = $"Hello there {Name}, age: {Age}";
result

When we feed this into our crude Roslyn based analyzer, we get this:

We have an Api Framework plugin doing this. It takes the analyzation results and does some code generation. This way we get OpenAPI schema for the given notebook and can execute the endpoint through a web request:

Before running the Notebook, we use CSharpKernel.SetValueAsync to set the arguments.