[Docs] Add to the roadmap: a good story for using Mojo from Python packages

[paugier]

Where is the problem?

https://docs.modular.com/mojo/roadmap.html

What can we do better?

This issue is just about adding a new item to the roadmap :-) about building a good story for using Mojo from Python packages (as also mentioned in #1161 and #546, see also https://github.com/ihnorton/mojo-ffi).

I discuss this point in my blog post on "Mojo: the point of view of a researcher using Python" and I identify this potential feature as very important to foster the adoption of Mojo among the Python developers.

I write here: "Enabling the compilation of Python extensions directly from Mojo code, perhaps through a @export decorator and a command like mojo build-py-ext, appears to be a feasible future prospect. Using the new HPy API would be a logical choice. Notably, this potential feature is currently absent from the Mojo website. Nevertheless, gaining insights from the Mojo team regarding their perspective on this feature would be interesting, as it is crucial for enhancing adoption within the Python community."

The simplest and most important approach would be to use Mojo SDK to produce a compiled Python extension usable without Mojo SDK.

Using the HPy Python API would be a huge advantage for Mojo. In particular, a Python extension created with Mojo would be usable by all recent CPython versions (without recompilation!) and would be efficient for all alternative Python interpreters supporting HPy (currently PyPy and GraalPy).

Then, supporting a JIT mode could be done through other tools in Python (something like Transonic (Disclaimer: I'm the author of Transonic.)) Anyway, this is a detail and of course does not need to be mentioned in Mojo FAQ. I'm only trying to explain why I think the most important thing is to be able to produce a Python extension from Mojo code with AOT compilation.

Note that only supporting conversions of standard Python types in read only mode and Numpy arrays (inout) would already be very useful.

Anything else?

No response

[paugier]

Another hack to call Mojo from Python (here with ctypes): https://gist.github.com/richardkiss/4f1fc7e948ddb7e8dba1e9a907662073

[paugier]

Let's also mention PyO3, which is the equivalent of what I'm talking about for Rust : "You can use PyO3 to write a native Python module in Rust".

[paugier]

A related old Discord question: https://discord.com/channels/1087530497313357884/1151550585590517801

[jackos]

@scottamain @arthurevans FYI

[paugier]

@arthurevans 's answer on Discord: https://discord.com/channels/1087530497313357884/1183054627189559359/1184199742826225674

So, we'll add something to the roadmap. The short answer is that this is definitely a goal, but there are still a lot of things that need to mature in the core language before we work on this.

[JackKelly]

@paugier asked me to talk about my use-case in this issue. To make life as easy as possible for readers of this current issue, I'll quote myself 🙂:

I would love to be able to call libraries written in Mojo from CPython.

I'm about to start writing a library for high speed IO & compute for chunked, compressed, multi-dimensional data. Some of the library's users (including me) are ML folks. But many of the library's users aren't doing ML. And some users are on highly constrained compute environments, where installing new software requires many months of discussions with overworked and understaffed IT departments.

So, even though I'm excited about Mojo, I know I can't force users of my library to use Mojo.

If CPython can't call Mojo then Mojo becomes "viral" like the GPL license. Code that uses Mojo forces all downstream users of that code to also use Mojo. I think that will harm adoption of Mojo! And I'd love to see Mojo do well! I'd love to see a super simple way for CPython users to install and use libraries written in Mojo!

That said, I'm really excited to hear that this feature will be added to the roadmap! And I really appreciate that Mojo is still very young, and has come an amazing distance already! So I don't want to put undue pressure on Mojo's dev team! I'll be patient 🙂

[paugier]

@arthurevans Where can we read the related changes? I looked again at https://docs.modular.com/mojo/roadmap.html but couldn't find anything new.

[arthurevans]

@paugier Changes should show up on the site with the next Mojo release. The added text isn't much more than a stake in the ground. It says:

Calling Mojo from Python

Currently you can call Python code from Mojo, but not the reverse: you can't pass a Mojo callback to a Python function, or build a Python extension in Mojo. We want to support calling Mojo from Python, but we want to do it right and we need the core language to be more mature first.

[OpenCoderX]

As someone who writes python daily, this would be a killer feature. Python is amazing until you hit a hot spot that can't be optimized any further. I would love to extract that hot spot into a mojo function, import it and call it with the mojo equivalent of ctypes. The wait will be worth it, especially if the interface between python and mojo is easier to define than ctypes when calling C/CPP.