Problems passing a std::vector by reference though virtual functions using pybind11

Question

I have some C++ code being bound to Python via pybind11, and we are having a strange issue with passing std::vector by reference through a virtual method and having the changes to that std::vector persist outside that method.

What we want to do is provide a C++ pure virtual function that accepts a std::vector by reference, so that a Python class that overrides this method can modify that std::vector in place, handing it back to other C++ code. We are following the pybind11 documentation about making the STL code opaque, but in our test we are finding that the STL vector is properly modified and passed by reference when the STL vector is created in Python, but if we create the std::vector in C++ it is not.

Here is a self-contained example that demonstrates the issue. First the C++ pybind11 code, where we are creating an abstract base class A with a single pure virtual method A::func which we want to override from Python:

#include "pybind11/pybind11.h"
#include "pybind11/stl_bind.h"
#include "pybind11/stl.h"
#include "pybind11/functional.h"
#include "pybind11/operators.h"

#include <iostream>
#include <vector>

namespace py = pybind11;
using namespace pybind11::literals;

PYBIND11_MAKE_OPAQUE(std::vector<int>)

//------------------------------------------------------------------------------
// The C++ types and functions we're binding
//------------------------------------------------------------------------------
struct A {
  virtual void func(std::vector<int>& vec) const = 0;
};

// In this function the std::vector "x" is not modified by the call to a.func(x)
// The difference seems to be that in this function we create the std::vector
// in C++
void consumer(A& a) {
  std::vector<int> x;
  a.func(x);
  std::cerr << "consumer final size: " << x.size() << std::endl;
}

// Whereas here, with the std::vector<int> created in Python and passed in, the
// std::vector "x" is modified by the call to a.func(x).
// The only difference is we create the std::vector in Python and pass it in here.
void another_consumer(A& a, std::vector<int>& x) {
  std::cerr << "another_consumer initial size: " << x.size() << std::endl;
  a.func(x);
  std::cerr << "another_consumer final size  : " << x.size() << std::endl;
}

//------------------------------------------------------------------------------
// Trampoline class for A
//------------------------------------------------------------------------------
class PYB11TrampolineA: public A {
public:
  using A::A;
  virtual void func(std::vector<int>& vec) const override { PYBIND11_OVERLOAD_PURE(void, A, func, vec); }
};

//------------------------------------------------------------------------------
// Make the module
//------------------------------------------------------------------------------
PYBIND11_MODULE(example, m) {
  py::bind_vector<std::vector<int>>(m, "vector_of_int");

  {
    py::class_<A, PYB11TrampolineA> obj(m, "A");
    obj.def(py::init<>());
    obj.def("func", (void (A::*)(std::vector<int>&) const) &A::func, "vec"_a);
  }

  m.def("consumer", (void (*)(A&)) &consumer, "a"_a);
  m.def("another_consumer", (void (*)(A&, std::vector<int>&)) &another_consumer, "a"_a, "x"_a);
}

We have also created two C++ standalone functions: consumer and another_consumer, which show examples of trying to pass and modify a std::vector<int> through this interface. In the case of consumer, this will fail, and it acts as though the argument to A::func is being passed by value. However, when we create the std::vector<int> in Python and pass it into another_consumer, things proceed as expected and the vector<int> is modified in place by A::func. Here's the Python code demonstrating the difference (assuming the above C++ is compiled to a module called example:

from example import *

class B(A):
    def __init__(self):
        A.__init__(self)
        return
    def func(self, vec):
        print "B.func START: ", vec
        vec.append(-1)
        print "B.func STOP : ", vec
        return

b = B()
print "--------------------------------------------------------------------------------"
print "consumer(b) -- This one seems to fail to pass back the modified std::vector<int> from B.func"
consumer(b)
print "--------------------------------------------------------------------------------"
print "another_consumer(b, x) -- This one works as expected"
x = vector_of_int()
another_consumer(b, x)
print "x : ", x

Executing the Python example results in:

--------------------------------------------------------------------------------
consumer(b) -- This one seems to fail to pass back the modified std::vector<int> from B.func
B.func START:  vector_of_int[]
B.func STOP :  vector_of_int[-1]
consumer final size: 0        
--------------------------------------------------------------------------------
another_consumer(b, x) -- This one works as expected
another_consumer initial size: 0
B.func START:  vector_of_int[]
B.func STOP :  vector_of_int[-1]
another_consumer final size  : 1
x :  vector_of_int[-1]            

So what are we misunderstanding here? Why does the first example, the function consumer, fail to have it's local copy of the std::vector modified in place and returned by reference from the Python method B.func?

Answer 1

The following is probably unsatisfactory, but it's such an easy "cheat" and gets you going, that I figured I might as well post it. As said in the comment above, the second case works b/c the python proxy object is found to already exist and is thus re-used. You can play that game yourself, by changing the trampoline code to:

class PYB11TrampolineA: public A {
public:
  using A::A;
  virtual void func(std::vector<int>& vec) const override { 
    py::object dummy = py::cast(&vec);   // force re-use in the following call
    PYBIND11_OVERLOAD_PURE(void, A, func, vec); 
  }
};

and then both consumer functions will work as expected.

EDIT: For that matter, since the problem is in the conversion of arguments handed to the callable (here: B.func), and since that is a Python, not C++, interface, you can alternatively simply pass &vec in the first place:

class PYB11TrampolineA: public A {
public:
  using A::A;
  virtual void func(std::vector<int>& vec) const override { 
    PYBIND11_OVERLOAD_PURE(void, A, func, &vec); 
  }
};

Internally to pybind11, it amounts to the same thing.

EDIT2: Found the cause. It's in pybind11/2114.h, line 2114: the use of std::forward removes the reference and the motivation appears to be preventing creating python proxies to temporaries. It'd be nice to have a specialization for objects-by-reference.