Is it possible to define a custom (cell) magic

[hanslovsky]

This is more of an inquiry/research question: I recently started playing around with Jep a bit and found it quite useful for combining the Java image processing library ImgLib2 and Kotlin extensions/operators with NumPy ndarray in native CPython. This all works with shared memory, really cool! I learned about library descriptors, which I could use to add Jep and a Python interpreter to the Kernel. Users could then call

interpreter.exec

and similar functions to execute their Python code.

Then I thought, maybe I can mark an entire cell as Python code via some custom cell magic, and the magic handler would then pass the Python code to the Python interpreter object that the Kernel holds. To make this work, I would need some way (that may or may not exist) to add a custom cell magic. Before I spend more time on it, I would like to know if something like that would be possible, in principle, or if I would be wasting my time.

Thank you!

[ileasile]

Hello @hanslovsky! Currently the list of magics is fixed and not extensible, but your idea is really cool. I think that I can implement even a more powerful way of doing it — custom code preprocessor which takes the whole cell code before the execution and can change it in any way. In your case it will wrap the cell code to the interpreter.exec() call.

[hanslovsky]

Thank you for the fast response and addressing this so quickly! This is much more than I could have hoped for and I am excited to see what will come out of it.

[altavir]

The problem with loading python code is that we will also have to somehow load python code completion. The similar thing was done in beakerx (though they just used different kernels for that).

[ileasile]

Custom completion contributors (as well as error reporters) are also not too hard to add via the API. I'll do it in a separate PR, I think

[hanslovsky]

The problem with loading python code is that we will also have to somehow load python code completion. The similar thing was done in beakerx (though they just used different kernels for that).

Is that within the scope of this feature? My inquiry (and understanding of the proposed solution) was more in the sense that the Kotlin kernel exposed a way to add a custom processor that I could use for a Python cell. My custom preprocessor would not be part of the official kernel (at least initially), so I don't think there would be a need to provide code completion.

[altavir]

If you do not need code completion and highlighting, then you actually do not need the cell magic you can do something like that:

val res = withPython("""
  callSomePythonFunction()
""")

[hanslovsky]

Thanks for addressing this so swiftly @ileasile
Can you share an example how I would add this to a library descriptor? I am not familiar the code base so it is hard for me to figure it out from the diff alone.

Thanks

[ileasile]

Hi! It is an advanced feature. We have a common concern not to add support of such features for JSON descriptors. But you can support it in your library/wrapper. First, set up the integration for your lib as described here: https://github.com/Kotlin/kotlin-jupyter/blob/master/docs/libraries.md#integration-using-kotlin-api

Then, add preprocessCode call to your integration class, it will be something like this for you:

@JupyterLibrary
internal class Integration : JupyterIntegration() {
    
    override fun Builder.onLoaded() {
        preprocessCode { code -> 
            if (code.startsWith("%%python")) { wrapPythonCode(code.removePrefix("%%python")) }
            else code
        }
    }
}

[ileasile]

BUT - I didn't think about it - there is actually a hacky way to go with JSON descriptor:

{
    "init": [
        "USE { preprocessCode { code -> if (code.startsWith(\"%%python\")) { wrapPythonCode(code.removePrefix(\"%%python\")) } else code } }"
    ]
}

But I don't recommend it :)

[hanslovsky]

I just tested the discouraged way by adding to the descriptor and it works beautifully! I only had to make the lambda return an (empty) string:

"USE { preprocessCode { code -> if (code.startsWith(\"%%python\")) { python.exec(code.removePrefix(\"%%python\")); \"\" } else code } }"

[ileasile]

Well, I'm glad that your problem is solved

[hanslovsky]

I was also able to create a library definition the proper way (example notebooks in the repo). I am really happy with how this turned out. Thanks again!