Using julia shared libraries as python C exstensions

[SimonDanisch]

I will try to post details of my progress here, in the hope to have a few more eyes looking at the problems I encounter.
Here is a start: https://discourse.julialang.org/t/calling-julia-shared-library-from-python-linking-libraries/9169

Let's hope we can do this quickly, this feature would be so awesome! :)

[SimonDanisch]

PR: #26

[tkf]

Hello PackageCompiler.jl devs, thanks for the awesome work! I find this issue interesting so allow me to join in. I'm a member of PyJulia so I may be biased 😉 (but try not to).

I think the biggest pros of loading library generated by PackageCompiler.jl is that it can release Python's Global Interpreter Lock (GIL). It means that you can run Python code and Julia code in parallel. However, the flip side of this is that you can't use PyCall (at the moment) for passing complex objects in between Julia and Python. But there are lot you can do with simple C types. (I guess something like Apache Arrow would help it too.) Another consequence is that it will not have the problem with precompilation cache for using PyCall.jl JuliaLang/julia#28518. On the other hand, PyJulia uses PyCall so it inherits all of PyCall's merits and restrictions.

Extending this line of thinking, I think the usage of the library generated by PackageCompiler.jl may be better focusing on C type-based interface rather than strongly coupling to Python C-API since using PyCall would be the easiest choice for this purpose. It means to not link against libpython (i.e., don't make it a "Python C extension" in a narrow sense) and use ctypes or cffi. Armin Ronacher discussed it in various talks like: https://www.youtube.com/watch?v=yhiHmBE9fNU (and https://www.youtube.com/watch?v=zmtHaZG7pPc&t=1095; I couldn't find a good readable material but this is close: https://blog.sentry.io/2017/11/14/evolving-our-rust-with-milksnake). The main point is that the distribution is going to be much easier since you don't need to build against various versions of libpython. For a working ctypes example, see the first link in #144. It would be nice to have an example of distributable Python pacakge combining PackageCompiler.jl and (maybe) milksnake (though I've never used it). It would allow Python users to just pip install some package without knowing Julia is working in the background.

[kdheepak]

We have a working example of this in PR #261.

[ViralBShah]

Would be nice to see if some of the PRs here can be revived (perhaps restarted?) and made to work.

[KristofferC]

This feels kind of stale. If this is still of interest, please open a new issue with updated information.