pynng should have option to use existing nng installation

[mc-allen]

Im my use case, we want to build NNG separately for use in both C/C++ and Python, and have pynng build bindings against that common installation. Currently it doesn't seem like there is a way to do this.

[codypiersall]

Thanks for filing an issue for this. This is a use case I'm interested in supporting, but it will have to wait until after #4 is finished.

First, let me say what is already possible. If you clone this repository, you can then clone nng, build a static library, and link against that library. You don't have to use the version that gets automatically installed whenever this library is installed. You could use that same library to link with C/C++ programs.

The challenge here is that the way pynng is built right now, it is required to build nng as a static library, and then link the Python extension module with that. If I'm understanding correctly, you would like to build nng as a shared library, and have pynng use the shared library. This is a valid use case, but it will take a little bit of work to get it actually supported. I still want the default build to be against a static nng of a known version on all platforms, but using a shared library should be supported for "advanced" use cases.

I guess it would probably mean editing the setup.py script to take an extra parameter, something like --use-shared-lib, and then using the ABI mode of cffi instead of the current API mode when that flag is present. I'm not really sure how much work that is, but it should be possible without needing to change the entire world.

[mc-allen]

I actually want to use a static library that I have built from source, with my preferred CMake flags and compiler options. My build also includes things like mbedtls, etc. I'd like pynng to use my version of nng library and header files, and not try to compile its own. Kind of a "bring your own artifacts" model. From my current investigation, that doesn't seem to be possible now, unless I am missing something.

[codypiersall]

Ah, got it. That's much easier to do, actually!

It is possible to do this now in a pretty inelegant way. The setup script looks to find the lib at ./nng/build/libnng.a (on Unix). If there is already a library there, the script will use it. So what you can do to get this working now would be to put a symlink to the file in that location, and the setup script will use it.

Note that on Unix platforms you must ensure that the static library gets built with the -fPIC flag, since a Python extension module is just a shared library in disguise.

So here's a pseudo worfklow that you can do:

cd ~
git clone https://github.com/nanomsg/nng
cd nng
# build the shared library, ensuring the `-fPIC` flag is set
cd ~
git clone https://github.com/codypiersall/pynng
cd pynng
ln -s ~/nng nng
pip install -e .

If you're using Windows, it's a similar story, but the library path relative to this repository is .\nng\build\Release\nng.lib, and symlinks don't work without admin privileges.

This is wayyyyy less than ideal. It shouldn't be hard to support a flag to the setup script to specify the path to the static lib. I'll look into this.

[mc-allen]

Any news on this ticket?

[codypiersall]

Thanks for the ping @mc-allen. I'm hoping to support this soonish -- next couple weeks. Springs are usually pretty busy for me though: gotta fight with my mower pretty regularly.

[codypiersall]

I've got an implementation of this in the custom-nng-build branch. I think the API is more-or-less okay, but the implementation feels pretty hacky (parsing args I care about and then replacing sys.argv so setuptools doesn't die).

To link to a custom lib, the setup script grew two arguments, which either must both be given or both absent: --nng-lib and --nng-include-dir. --nng-lib is the path to the nng library file, and --nng-include-dir is the path to the include directory for nng. (I'm thinking about making that option, well, optional, since if nng is actually installed the compiler should be able to pick up the right include directory anyway...)

tldr; run the setup script like this to install with a custom nng library:

python setup.py --nng-lib=/path/to/libnng.a --nng-include-dir=/path/to/include install

Any feedback about the usage or implementation would be appreciated. I'm going to add some docs before merging it into master, but that'll probably be another handful of days.

[wylderkeane]

You could probably implement the install of the library as a separate command by subclassing distutils.cmd.Command, and then invoke that command in your build_py subclass. This way you can provide lots of options for customizing the NNG library used (or built). This page describes the technique I'm referring to, and provides a simple example.

By the comments in the file it appears you'd like to clean some of that custom builder stuff up anyway, and this may be a nice way to do it. The other nice property of doing it this way is you can set some default arguments for the NNG library build in the setup.cfg file and allow users to edit them for their own purposes.

[codypiersall]

Thanks for the pointer @wylderkeane. I'll take a look through the linked page in the next couple days.

You could probably implement the install of the library as a separate command by subclassing distutils.cmd.Command, and then invoke that command in your build_py subclass

That seems like a route worth checking out.

you can set some default arguments for the NNG library build in the setup.cfg file and allow users to edit them for their own purposes.

The use case that I think @mc-allen has is that he already has built an nng lib, totally outside of pynng, and would like to use that; in which case having the flexibility in the setup.cfg wouldn't be very helpful.

[wylderkeane]

What I'm suggesting is the setup.cfg file would allow the same customization as the argument flags, including the ability to specify a prebuilt version of nng lib. You get this for free by defining the new command with a set of user_options, it's just another way to specify them. I just wanted to point it out in case there were some other defaults or parameters that were magically derived in the install section that you wanted to expose (like to build as a shared lib or static lib).

[codypiersall]

@wylderkeane Ah, yeah, I get what you're saying now; using a custom command for sure is a better strategy. I like it! That will be the route I'll pursue.

[zakharov]

I have tried to use build pynng with an existing shared library of nng, which is compiled with mbedTLS support. Seem to work fine. Do you have future plans to work on this issue? To merge custom-nng-build into the master branch?

Thanks.

[codypiersall]

Hi @zakharov,

Thanks for the interest and confirming it works. I'm planning to merge either custom-nng-build or something like it, but I need to revisit what I did and write docs for it. I probably won't merge something until around the holidays, when I will hopefully have more time to work on pynng.

[leonardp]

Hello!
I want to package this module for NixOS and ran into related issues. I can get it to work but it's kinda hacky..
Is there any update on this?