Binary with dynamic dependency

[alecandido]

Essentially, the problem is the one described by the title:

• I chose as binding type bin, but my binary is dynamically linked to a library
• the library is shipped in the wheel, but:
  • the name is mangled (so even setting LD_LIBRARY_PATH manually is not working)
  • rpath is wrong

The full reproduction is already spelled out in another issue: pypa/auditwheel#396

They already closed the issue as duplicated, since the problem on their side is the same of pypa/auditwheel#340, where they also provide a workaround:

The only solution I see is to move the scripts to platlib, where we know its relative path to the .libs folder, and place a wrapper in its place that calls os.execv on the actual binary.

I wonder if it's possible (i.e. quick enough) to implement the workaroud in maturin.

[messense]

I think it's possible to implement the workaround.

• We only want to do it for executables that are not manylinux/musllinux compliant, which means it's fine to put them in scripts if they only dynamic link to allowed dependencies.
• To be portable I think it's better to use console_scripts as wrapper instead of placing wrapper in scripts because it's complicated on Windows (You need to add .py suffix to instruct Windows to launch it with Python but then you can't make sure it uses the right Python version).

I'm not sure it's a quick job, you'll have to look into it. It might be much quicker if you can change to static link your dependencies.

[alecandido]

It might be much quicker if you can change to static link your dependencies.

Yes, this is definitely the way to go in the short term (also because I'm not familiar with your codebase, it would take me while to make a PR).
But the option to ship compiled binaries is interesting, so I decided to file the issue in any case. If not for this one (it is also a relatively small library), it might be useful later on :)

[messense]

You can also turn the binary into a extension module instead and add a console_script entrypoint that calls a function of the extension module to make a cli, this way you can still use dynamic dependency.

[alecandido]

• To be portable I think it's better to use console_scripts as wrapper instead of placing wrapper in scripts because it's complicated on Windows (You need to add .py suffix to instruct Windows to launch it with Python but then you can't make sure it uses the right Python version)

This is not a problem for me, because the CLI I'm trying to package has no Python at all.
It is part of the same suite of a library for which I provided Python bindings, and packaged with maturin. So, I'm trying to make a CLI only package (for simplicity, since lib and CLI are two different Rust crates), and then I'll make it a dependency for the former one.

You can also turn the binary into a extension module instead and add a console_script entrypoint that calls a function of the extension module to make a cli, this way you can still use dynamic dependency.

This is actually a pretty simple option, I'm considering it. But, if reasonable, I'd still prefer static compilation (since there is no Python at the moment, if possible I'd keep Python free).

[messense]

Well, if you are distributing binaries using Python wheels, you are not really Python free though.

[alecandido]

Agreed, you have a point.

But one thing is to depend on Python infrastructure for packaging, while here running does not involve the interpreter at all.

Of course, the reason is not that it might not be available (if you are installing with pip you have an interpreter), but just because it is the minimal option: the CLI is already running on its own, if I can distribute as is there is no extra layer I have to maintain.

[messense]

I think the most obvious benefits for Python free binary is startup time, but if we are going to implement the workaround by moving the executable in platlib and place a wrapper (written in Python obviously) in its place that calls os.execv on the actual binary, it just goes back to start Python interpreter first which loses the performance benefits.

Unless we write the wrapper in sh on Unix or batch/powershell on Windows (which might also have portability issues consider all the platforms maturin supports), you can not avoid using Python.

[alecandido]

With that, I meant that my favorite option is still static compilation, in this case.

I agree that the workaround requires Python, and as you said that's part of the game. I also agree that using console_scripts is a good idea, since we'll leverage part of the interfacing work done somewhere else.

So, I'm going to aim at statically compiling my binary. But, for the future, I'd like to implement (or see implemented) also the workaround, such that bin bindings will work out of the box also for dynamic dependencies (with all the required trade-offs).