Function arguments by reference

[ianhbell]

I know dealing with references is never easy when wrapping low-level languages in high-level languages (I've fought with this in SWIG and Cython), but how would I get a dummy function like this working:

void plus_one_inplace(int &i) { i++; }

I tried:

PYBIND11_PLUGIN(ex) {
    py::module m("ex", "example");
    m.def("plus_one_inplace", &plus_one_inplace, "add one inplace");
    return m.ptr();
}

but when I do

i=1
ex.plus_one_inplace(i)

i is still equal to 1

[wjakob]

You can't. Try writing a function like that in Python and see if you can get it to work ;)

[ianhbell]

But passing arguments by reference is a very common idiom in C/C++. So I think it would be nice if the docs at least said:
a) You can't do it
b) Provided a small example showing how to work around this limitation

[wjakob]

Yeah, adding it to the FAQ now. I fear that there is no workaround in this case -- whatever you do, it has to make sense to the language itself and ints are immutable in Python.

[ianhbell]

Of course the obvious solution is to wrap everything in a class, which is then stateful, but I am trying to avoid that if I can

[wjakob]

faq added

[ianhbell]

Thanks for the FAQ. The "solution" for SWIG if you are using references as a means of returning multiple values is to use a typemap which returns a tuple rather than a single double. How would one go about that doing something like that in pybind11 without changing the C++ code?

[wjakob]

write a lambda function which calls your function and then bind that lambda function -- that kind of stuff is shown in the advanced section of the docs. There, simply return a tuple or pair, etc.

[ianhbell]

Us folks coming from the pre-C++11 world might need a little bit of assistance getting up to speed with this concept. A simple example in the advanced section would be appreciated.

[wjakob]

If you read through the advanced section, I think that this stuff should be pretty clear. If not, I can add something.

[ianhbell]

Yes, a simple example would be illustrative. For instance, let's suppose I have a trivial function (for illustrative purposes) like this:

void set_values(double &v1, double &v2){
    v1 = 42;
    v2 = 43;
}

how would I wrap that with a lambda to return the output values?

[wjakob]

updated -- please note that I generally don't have time for this kind of C++11 tutoring ;)

[ianhbell]

Fair enough.

I'm very, very appreciative of what you have done with pybind11. As a heavy user of both C++ and python, this is the nicest solution that I have seen thus far for bridging the gap. I've previously used SWIG and cython extensively, and I am considering moving all my interfaces to pybind11

The only gripe that I have is that C++11 is required, which means no visual studio compilation for the standard python distributions, and for windows you need to fight a bit with MINGW to get it to play nicely, especially when you link against static libraries compiled with MINGW. Getting the compilation flags right is a real pain.

[wjakob]

Compilation should "just" work (even with Visual Studio/MINGW) but obviously you'll need to have a C++11 capable version. (Visual studio 2015, not sure for MinGW but there are people who successfully used it)

[ianhbell]

Right, but the issues are on windows (as usual). Python 2.7 is build with Visual Studio 9 in most distributions, and some 3.x versions are built with Visual Studio 10, neither of which support C++11. Therefore, for these python versions, you MUST use mingw since Visual Studio 15 doesn't generate binaries compatible with either VS9 or VS10.

[wjakob]

Not true -- try it with MSVC 2015, you'll be surprised ;). pybind11 is very careful to do its own internal management of data structures to avoid ABI incompatibilities.

[wjakob]

(It's perfectly feasible to have to DLLs talk to each other that are built with different compilers/C libraries. The things to watch out for are not to free() a pointer that is malloc()ed in the other, and so on..)

[ianhbell]

?? My understanding (perhaps incorrect), is that the python static library for py27 on windows is built with VS9, and therefore if you were to try to link against it in VS15, you would not be able to due to ABI incompatibilities. I am certainly willing to try with VS15. I do have MINGW working though, so that's an ok backup plan.

If I am wrong, this would be a most interesting thing to have learned.

[wjakob]

Look at the continuous integration bots -- that's the exact setup they use (default VS9-based distribution of Python together with pybind11 built with MSVC 2015). They pass all testcases, so that should give you some level of confidence ;)

[wjakob]

Another proposed addition to the FAQ:

Working with ancient Visual Studio 2009 builds on Windows
=========================================================

The official Windows distributions of Python are compiled using truly ancient
versions of Visual Studio that lack good C++11 support. Some users implicitly
assume that it would be impossible to load a plugin built with Visual Studio
2015 into a Python distribution that was compiled using Visual Studio 2009.
However, no such issue exists: it's perfectly legitimate to interface DLLs that
are built with different compilers and/or C libraries. Common gotchas to watch
out for involve not ``free()``-ing memory region that that were ``malloc()``-ed
in another shared library, using data structures with incompatible ABIs, and so
on. pybind11 is very careful not to make these types of mistakes.

[ianhbell]

Wow. This is one of the most useful discussions I have had related to python in a long, long time.

[jieli-matrix]

Here I'd like to give a binding example of function plus_one_inplace. cc: @ianhbell
Suppose you have a c++ function with arguments by reference as follows,

void plus_one_inplace(int &i) { i++; }

you can bind it with lambda function in pybind like this

PYBIND11_MODULE(_core, m) {
    m.def("plus_one_inplace", [](int i){plus_one_inplace(i); return i;}, "add one inplace");
}

and import _core in python

import _core
i = 1
i = m.plus_one_inplace(i)
i

and then you will get i=2.

as @wjakob says, ints are immutable types in python so you need to wrap in lambda function and return value.

Extend reference type from int to vector, you may try opaque in pybind to avoid unnecessary copy movement, and refer to the link.

[xieyuheng]

@jieli-matrix
Thanks for your example, that's helpful :)