-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is clone pattern supported with classes overloaded in python? #1049
Comments
Isn't the whole point of a virtual Why don't you just add |
Yes, this is what I did initially. Unfortunately it doesn't work:
That is what I can't understand, actually. |
When exactly does that happen? You have implemented a non-pure |
Wait, I'm now wondering if |
@yesint Can you post a complete code example (both C++ and Python sides) which reproduces the error that you're seeing? |
@dean0x7d Here is minimal example to reproduce:
And the python code to test:
The error:
|
Unless I call clone() this cast gives me pointer to correct object, which is able to call python-derived func():
|
Correct me if I'm wrong (I don't know internals of pybind11) but it seems that when clone() is called only C++ A_trampoline object is created but not the B instance on the python side. As a result the trampoline redirects the virtual call to nothing ending up in calling pure-virtual A::func(). So the idea as far as I understand is to create B instance in python interpreter as usual and access it from A_trampoline::clone instead of allocating A_trampoline at C++ side. But I have no idea how to implement this. |
Oh, yes, of course! You are right; I'd failed to notice that complexity! One option is then of course to override the Another option is to use the type of the actual Python object like this: virtual A_trampoline* clone() const {
auto this_object = py::cast(this);
return py::cast<A_trampoline*>(this_object.get_type()(this_object).release());
} and then add .def(py::init_alias<const A_trampoline &>()) The downside of this approach is that when you implement EDIT: |
Wait, actually, this might be nastier than it seemed at first. Because ... where is the actual Python object now stored (including its attributes)? I guess this either means something's leaking, or that call to |
Well, this black magic works :) At least the program now behaves as expected.
Gives:
|
Yeah, there's the thing with Another option would to try and see if you can replace this copy-constructor thing with Python's Edit: just doing this works, btw: def __init__(self, *args):
A.__init__(self, *args) |
But also, I'm still not convinced that this |
Well, it seems that the attributes are not copied and have to be assigned in python-side "copy-constructor":
I'm not familiar with deepcopy() enough to figure out how to use it in |
Btw, both copy() and deepcopy() fails on B:
|
I don't really know either, but you can implement those by using
I do not really know what that error message is about. But the memory allocation and initialization of pybind11 is not that easy, so I guess the integration with actual Python objects is not that clear-cut :-/ |
Yes, that's exactly it.
One issue here is that the reference of the newly created object is leaked following Given the requirement to keep the Python state alive, I think the best solution here is to make #include <pybind11/pybind11.h>
namespace py = pybind11;
class A {
public:
A() = default;
A(const A&) {}
virtual ~A() = default;
virtual void func() = 0;
virtual std::shared_ptr<A> clone() const = 0;
};
class A_trampoline: public A {
public:
using A::A;
A_trampoline(const A& a) : A(a) {}
void func() override { PYBIND11_OVERLOAD_PURE(void, A, func,); }
std::shared_ptr<A> clone() const override {
auto self = py::cast(this);
auto cloned = self.attr("clone")();
auto keep_python_state_alive = std::make_shared<py::object>(cloned);
auto ptr = cloned.cast<A_trampoline*>();
// aliasing shared_ptr: points to `A_trampoline* ptr` but refcounts the Python object
return std::shared_ptr<A>(keep_python_state_alive, ptr);
}
};
PYBIND11_MODULE(example, m) {
py::class_<A, A_trampoline, std::shared_ptr<A>>(m, "A")
.def(py::init<>())
.def(py::init<const A&>())
.def("func", &A::func);
m.def("call", [](const std::shared_ptr<A>& inst){
inst->func();
inst->clone()->func();
});
} import example as m
class B(m.A):
def __init__(self):
m.A.__init__(self)
self.x = 0
def func(self):
print("hi!", self.x)
def clone(self):
print("clone")
# create a new object without initializing it
cloned = B.__new__(B)
# clone C++ state
m.A.__init__(cloned, self)
# clone Python state
cloned.__dict__.update(self.__dict__)
return cloned
inst = B()
inst.x = 1
m.call(inst) |
Wow! That is indeed much more involved that I thought. Thank you very much, @dean0x7d ! |
Nice, I had not idea about the aliasing constructor of an Now I'm just wondering, if you want to make this transparent to the Python side, could you maybe still move the
part into the |
@YannickJadoul Yes, that part could definitely be moved into C++. At that point, I guess pybind11 might be able to offer some utility to make that whole procedure easier and it might also be interesting to consider plugging into Python's I think this should be documented but not exactly in the complicated form above. This needs to cook a bit more, to be easier to use, but I don't think I'll have time to develop it much more. If anyone would like to look into this and refine it further, feel free. |
I am currently trying to implement a very similar workflow, and I've found a solution that nearly does what is described in this issue (with a unique pointer instead of a shared pointer), except that the reference counting somehow goes wrong and I don't understand why. (If I should rather create a new issue, do tell me, but I feel like this belongs here) The basic idea is the same as above, I have a virtual cloneable C++ class and its Python trampoline (full code at the end): struct virtual_base
{
virtual void call() = 0;
virtual std::unique_ptr<virtual_base> clone() const = 0;
};
struct py_virtual_base
: virtual_base
{
void call() override;
std::unique_ptr<virtual_base> clone() const override;
}; And then define a child class on the Python side: class child_virtual(po.virtual_base):
def __init__(self):
super().__init__()
def call(self):
print("Called child class") Now, as far as my understanding of the pybind11 object model goes, this will create a Python object that owns an instance of struct OriginalInstance
{
py::handle pythonObject;
~OriginalInstance()
{
pythonObject.dec_ref();
}
};
/*
* If the shared pointer is empty, this object is the original object owned
* by Python and the Python handle can be acquired by:
* py::cast(static_cast<ChildPy const *>(this))
*
* Copied instances will refer to the Python object handle via this member.
* By only storing this member in copied instances, but not in the original
* instance, we avoid a memory cycle and ensure clean destruction.
*/
std::shared_ptr<OriginalInstance> m_originalInstance; This design allows using unique pointers and still avoids memory loops. The Python object will wait for the C++ objects to deallocate. #include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <iostream>
#include <memory>
namespace py = pybind11;
/*
* py_virtual_base is the C++ representation for objects created in Python.
* One challenge about these classes is that they cannot be easily copied or
* moved in memory, as the clone will lose the relation to the Python object.
* This class has a clone_impl() method that child classes can use for cloning
* the object and at the same time storing a reference to the original Python
* object.
* The template parameters ChildCpp and ChildPy implement a CRT-like pattern,
* split into a C++ class and a Python trampoline class as documented here:
* https://pybind11.readthedocs.io/en/stable/advanced/classes.html?highlight=trampoline#overriding-virtual-functions-in-python
*
* A typical child instantiation would look like:
* struct ChildPy : ChildCpp, ClonableTrampoline<ChildCpp, ChildPy>;
*/
template <typename ChildCpp, typename ChildPy>
struct ClonableTrampoline
{
struct OriginalInstance
{
py::handle pythonObject;
~OriginalInstance()
{
pythonObject.dec_ref();
}
};
/*
* If the shared pointer is empty, this object is the original object owned
* by Python and the Python handle can be acquired by:
* py::cast(static_cast<ChildPy const *>(this))
*
* Copied instances will refer to the Python object handle via this member.
* By only storing this member in copied instances, but not in the original
* instance, we avoid a memory cycle and ensure clean destruction.
*/
std::shared_ptr<OriginalInstance> m_originalInstance;
[[nodiscard]] py::handle get_python_handle() const
{
if (m_originalInstance)
{
std::cout << "Refcount "
<< m_originalInstance->pythonObject.ref_count()
<< std::endl;
return m_originalInstance->pythonObject;
}
else
{
auto self = static_cast<ChildPy const *>(this);
return py::cast(self);
}
}
template <typename Res, typename... Args>
Res call_virtual(std::string const &nameOfPythonMethod, Args &&...args)
{
py::gil_scoped_acquire gil;
auto ptr = get_python_handle().template cast<ChildCpp *>();
auto fun = py::get_override(ptr, nameOfPythonMethod.c_str());
if (!fun)
{
throw std::runtime_error(
"Virtual method not found. Did you define '" +
nameOfPythonMethod + "' as method in Python?");
}
auto res = fun(std::forward<Args>(args)...);
return py::detail::cast_safe<Res>(std::move(res));
}
[[nodiscard]] std::unique_ptr<ChildCpp> clone_impl() const
{
auto self = static_cast<ChildPy const *>(this);
if (m_originalInstance)
{
return std::make_unique<ChildPy>(*self);
}
else
{
OriginalInstance oi;
oi.pythonObject = py::cast(self);
// no idea why we would need this twice, but we do
oi.pythonObject.inc_ref();
oi.pythonObject.inc_ref();
auto res = std::make_unique<ChildPy>(*self);
res->m_originalInstance =
std::make_shared<OriginalInstance>(std::move(oi));
return res;
}
}
};
struct virtual_base
{
virtual void call() = 0;
virtual std::unique_ptr<virtual_base> clone() const = 0;
};
struct py_virtual_base
: virtual_base
, ClonableTrampoline<virtual_base, py_virtual_base>
{
void call() override
{
call_virtual<void>("call");
}
std::unique_ptr<virtual_base> clone() const override
{
return clone_impl();
}
};
void call_from_base(virtual_base &obj)
{
obj.call();
}
PYBIND11_MODULE(python_override, m)
{
py::class_<virtual_base, py_virtual_base>(m, "virtual_base")
.def(py::init<>())
.def("call", &virtual_base::call)
.def("clone", &virtual_base::clone);
m.def("call_from_base", &call_from_base);
} CMake: cmake_minimum_required(VERSION 3.12.0)
project(pybind11_test)
find_package(pybind11 2.9.1 REQUIRED)
pybind11_add_module(python_override main.cpp) And using this in Python: #!/usr/bin/env python3
import build.python_override as po
class child_virtual(po.virtual_base):
def __init__(self):
super().__init__()
def call(self):
print("Called child class")
def main():
obj = child_virtual()
obj.call()
po.call_from_base(obj)
obj_cloned = obj.clone()
print("Will delete object now")
del obj
print("Deleted object now")
obj_cloned.call()
po.call_from_base(obj_cloned)
main() Special note goes to the repeated // no idea why we would need this twice, but we do
oi.pythonObject.inc_ref();
oi.pythonObject.inc_ref(); The output of this is:
However, when increasing the refcount only once:
|
In C++ clone pattern is used in many APIs:
Suppose one extends class A in python using standard trampoline facilities and exposes it as usual:
When clone() is called inside work_on_many_instances() it creates an instance of A, which is an abstract class and, of course, fails. Unless I'm missing something obvious I can't figure out how to clone an object of derived class correctly. As far as I understand the trampoline should make a deep copy of existing python derived class instance on C++ side somehow but I have absolutely no idea how. Is it possible at all?
The text was updated successfully, but these errors were encountered: