Mixed language package with pure python fallback

[SonOfLilit]

I've encountered a use case that it seems lots of people have encountered. I would submit a PR but I have no idea where to start.

I have a package written in Python (in my case Kleenexp, a modern regex syntax) and part of it was ported to Rust for speed or other reasons, but the pure python fallback still exists. I'd like "pip install" to install a binary wheel if available, and otherwise try co compile it but just use the pure python fallback if there's no rust toolchain available (or anything else prevents compiling).

Here are some other people describing the same problem:
https://stackoverflow.com/questions/41778153/compiling-an-optional-cython-extension-only-when-possible-in-setup-py
pypa/packaging-problems#214

Setuptools has Extension.optional to address this:
https://docs.python.org/3/distutils/apiref.html#distutils.core.Extension

Ideally maturin would add a similar option.

Before Extension.optional was available, people added something like this to setup.py:

try:
    # try building with c code :
    setup(ext_modules=ext_modules, **setup_args)
except ext_errors as ex:
    log.warn(ex)
    log.warn("The C extension could not be compiled")

    ## Retry to install the module without C extensions :
    # Remove any previously defined build_ext command class.
    if 'build_ext' in setup_args['cmdclass']:
        del setup_args['cmdclass']['build_ext']

    # If this new 'setup' call don't fail, the module 
    # will be successfully installed, without the C extension :
    setup(**setup_args)
    log.info("Plain-Python installation succeeded.")

Parhaps a similar workaround could be made with the maturin build engine. However, I don't understand how to get setup_args and ext_modules or equivalent.

Another workaround, which I'm going to use until I can resolve this but am very unhappy with, is to have a python package kleenexp and a faster package kleenexp-native and add an optional feature to kleenexp so pip install kleenexp[native] would install both.

[messense]

Nowadays most platforms have pre-built wheels support, users do not need to compile from source. What platforms are you trying to support with pure python fallback?

[konstin]

I think the canonical solution is that fringe platforms that don't have manylinux support, pip install will pick the source distribution and compile on installing. The only caveat is that users need to install rust since cargo/rustc aren't on pypi (yet).

One thing you could try is publishing platform specific wheels and an sdist with maturin and under same name/version a pure python platform independent wheel. You'd need to test it but i think pip prefers binary wheels over platform independent ones. (In much speculative ideas: i think having an optional package using wasm instead of the c api with lower precendence than native wheel would be really cool)

Another workaround, which I'm going to use until I can resolve this but am very unhappy with, is to have a python package kleenexp and a faster package kleenexp-native and add an optional feature to kleenexp so pip install kleenexp[native] would install both.

Yep, i've seen a couple a packages that have an optional performance extra that install some native packages.

[SonOfLilit]

Nowadays most platforms have pre-built wheels support, users do not need to compile from source. What platforms are you trying to support with pure python fallback?

Those that I don't know about. I'm trying to promise 100% drop-in compatibility with the re standard library, so I can't have caveats like "except if you try to run it in a really weird platform".

I think the canonical solution is that fringe platforms that don't have manylinux support, pip install will pick the source distribution and compile on installing. The only caveat is that users need to install rust since cargo/rustc aren't on pypi (yet).

The canonical solution is the Extension.optional parameter. I found a better workaround, which is to drop maturin and use setuptools-rust that supports Extension.optional, but I still think this is a good feature to add (and if it's easy and I can get some guidance, I can submit a PR).

[konstin]

maturin being itself written in rust only works on rust-supported platforms and only has wheels for at least somewhat common platforms, so unfortunately "anything that a ported gcc can get python compiled on", while it would be cool, is out of scope for maturin (this is a general problem with rust extensions, see e.g. the complaints here about cryptography adding). If you really want 100% std compatibility i'm afraid you'll have to use setuptools or maybe even something custom.

[SonOfLilit]

Great, thanks, that makes sense, that just means my workaround is the best solution (assuming it's not easy to implement this in the python side of maturin). I'll submit a PR documenting this.

…

On Sun, 4 Sept 2022, 14:44 konstin, ***@***.***> wrote: maturin being itself written in rust only works on rust-supported platforms and only has wheels for at least somewhat common platforms, so unfortunately "anything that a ported gcc can get python compiled on", while it would be cool, is out of scope for maturin (this is a general problem with rust extensions, see e.g. the complaints here <pyca/cryptography#5771> about cryptography adding). If you really want 100% std compatibility i'm afraid you'll have to use setuptools or maybe even something custom. — Reply to this email directly, view it on GitHub <#1081 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADU7NZFHT3RKJC7M6RVG3V4SDTJANCNFSM6AAAAAAQEGCV7Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[messense]

assuming it's not easy to implement this in the python side of maturin

It'd require resolve #563 first, then make maturin pep517 non-fatal when it can't build the extension module if configured.

[konstin]

It'd require resolve #563 first, then make maturin pep517 non-fatal when it can't build the extension module if configured.

Wouldn't this require to have maturin's bootstrapping code detect rust doesn't work on the target platform and then instead build a dummy pure python maturin which in turn builds the python only wheel of the target package?

[messense]

Wouldn't this require to have maturin's bootstrapping code detect rust doesn't work on the target platform and then instead build a dummy pure python maturin which in turn builds the python only wheel of the target package?

Yes, if the platform can not bootstrap maturin we'd need this, otherwise if it only can't successfully compile the extension module we'd need to make maturin pep517 non-fatal.

Sounds like a lot of work.

[messense]

Closing as out of scope, better to use setuptools-rust instead for more control.