Functions¶

Before proceeding with this section, make sure that you are already familiar with the basics of binding functions and classes, as explained in First steps and Object-oriented code. The following guide is applicable to both free and member functions, i.e. methods in Python.

Return value policies¶

Python and C++ use fundamentally different ways of managing the memory and lifetime of objects managed by them. This can lead to issues when creating bindings for functions that return a non-trivial type. Just by looking at the type information, it is not clear whether Python should take charge of the returned value and eventually free its resources, or if this is handled on the C++ side. For this reason, pybind11 provides a several return value policy annotations that can be passed to the module_::def() and class_::def() functions. The default policy is return_value_policy::automatic.

Return value policies are tricky, and it’s very important to get them right. Just to illustrate what can go wrong, consider the following simple example:

/* Function declaration */
Data *get_data() { return _data; /* (pointer to a static data structure) */ }
...

/* Binding code */
m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python

What’s going on here? When get_data() is called from Python, the return value (a native C++ type) must be wrapped to turn it into a usable Python type. In this case, the default return value policy (return_value_policy::automatic) causes pybind11 to assume ownership of the static _data instance.

When Python’s garbage collector eventually deletes the Python wrapper, pybind11 will also attempt to delete the C++ instance (via operator delete()) due to the implied ownership. At this point, the entire application will come crashing down, though errors could also be more subtle and involve silent data corruption.

In the above example, the policy return_value_policy::reference should have been specified so that the global data instance is only referenced without any implied transfer of ownership, i.e.:

m.def("get_data", &get_data, py::return_value_policy::reference);

On the other hand, this is not the right policy for many other situations, where ignoring ownership could lead to resource leaks. As a developer using pybind11, it’s important to be familiar with the different return value policies, including which situation calls for which one of them. The following table provides an overview of available policies:

Return value policy	Description
`return_value_policy::take_ownership`	Reference an existing object (i.e. do not create a new copy) and take ownership. Python will call the destructor and delete operator when the object’s reference count reaches zero. Undefined behavior ensues when the C++ side does the same, or when the data was not dynamically allocated.
`return_value_policy::copy`	Create a new copy of the returned object, which will be owned by Python. This policy is comparably safe because the lifetimes of the two instances are decoupled.
`return_value_policy::move`	Use `std::move` to move the return value contents into a new instance that will be owned by Python. This policy is comparably safe because the lifetimes of the two instances (move source and destination) are decoupled.
`return_value_policy::reference`	Reference an existing object, but do not take ownership. The C++ side is responsible for managing the object’s lifetime and deallocating it when it is no longer used. Warning: undefined behavior will ensue when the C++ side deletes an object that is still referenced and used by Python.
`return_value_policy::reference_internal`	Indicates that the lifetime of the return value is tied to the lifetime of a parent object, namely the implicit `this`, or `self` argument of the called method or property. Internally, this policy works just like `return_value_policy::reference` but additionally applies a `keep_alive<0, 1>` call policy (described in the next section) that prevents the parent object from being garbage collected as long as the return value is referenced by Python. This is the default policy for property getters created via `def_property`, `def_readwrite`, etc.
`return_value_policy::automatic`	This policy falls back to the policy `return_value_policy::take_ownership` when the return value is a pointer. Otherwise, it uses `return_value_policy::move` or `return_value_policy::copy` for rvalue and lvalue references, respectively. See above for a description of what all of these different policies do. This is the default policy for `py::class_`-wrapped types.
`return_value_policy::automatic_reference`	As above, but use policy `return_value_policy::reference` when the return value is a pointer. This is the default conversion policy for function arguments when calling Python functions manually from C++ code (i.e. via `handle::operator()`) and the casters in `pybind11/stl.h`. You probably won’t need to use this explicitly.

Return value policies can also be applied to properties:

class_<MyClass>(m, "MyClass")
    .def_property("data", &MyClass::getData, &MyClass::setData,
                  py::return_value_policy::copy);

Technically, the code above applies the policy to both the getter and the setter function, however, the setter doesn’t really care about return value policies which makes this a convenient terse syntax. Alternatively, targeted arguments can be passed through the cpp_function constructor:

class_<MyClass>(m, "MyClass")
    .def_property("data",
        py::cpp_function(&MyClass::getData, py::return_value_policy::copy),
        py::cpp_function(&MyClass::setData)
    );

Warning

Code with invalid return value policies might access uninitialized memory or free data structures multiple times, which can lead to hard-to-debug non-determinism and segmentation faults, hence it is worth spending the time to understand all the different options in the table above.

Note

One important aspect of the above policies is that they only apply to instances which pybind11 has not seen before, in which case the policy clarifies essential questions about the return value’s lifetime and ownership. When pybind11 knows the instance already (as identified by its type and address in memory), it will return the existing Python object wrapper rather than creating a new copy.

Note

The next section on Additional call policies discusses call policies that can be specified in addition to a return value policy from the list above. Call policies indicate reference relationships that can involve both return values and parameters of functions.

Note

As an alternative to elaborate call policies and lifetime management logic, consider using smart pointers (see the section on Custom smart pointers for details). Smart pointers can tell whether an object is still referenced from C++ or Python, which generally eliminates the kinds of inconsistencies that can lead to crashes or undefined behavior. For functions returning smart pointers, it is not necessary to specify a return value policy.

Additional call policies¶

In addition to the above return value policies, further call policies can be specified to indicate dependencies between parameters or ensure a certain state for the function call.

Keep alive¶

In general, this policy is required when the C++ object is any kind of container and another object is being added to the container. keep_alive<Nurse, Patient> indicates that the argument with index Patient should be kept alive at least until the argument with index Nurse is freed by the garbage collector. Argument indices start at one, while zero refers to the return value. For methods, index 1 refers to the implicit this pointer, while regular arguments begin at index 2. Arbitrarily many call policies can be specified. When a Nurse with value None is detected at runtime, the call policy does nothing.

When the nurse is not a pybind11-registered type, the implementation internally relies on the ability to create a weak reference to the nurse object. When the nurse object is not a pybind11-registered type and does not support weak references, an exception will be thrown.

If you use an incorrect argument index, you will get a RuntimeError saying Could not activate keep_alive!. You should review the indices you’re using.

Consider the following example: here, the binding code for a list append operation ties the lifetime of the newly added element to the underlying container:

py::class_<List>(m, "List")
    .def("append", &List::append, py::keep_alive<1, 2>());

For consistency, the argument indexing is identical for constructors. Index 1 still refers to the implicit this pointer, i.e. the object which is being constructed. Index 0 refers to the return type which is presumed to be void when a constructor is viewed like a function. The following example ties the lifetime of the constructor element to the constructed object:

py::class_<Nurse>(m, "Nurse")
    .def(py::init<Patient &>(), py::keep_alive<1, 2>());

Note

keep_alive is analogous to the with_custodian_and_ward (if Nurse, Patient != 0) and with_custodian_and_ward_postcall (if Nurse/Patient == 0) policies from Boost.Python.

Call guard¶

The call_guard<T> policy allows any scope guard type T to be placed around the function call. For example, this definition:

m.def("foo", foo, py::call_guard<T>());

is equivalent to the following pseudocode:

m.def("foo", [](args...) {
    T scope_guard;
    return foo(args...); // forwarded arguments
});

The only requirement is that T is default-constructible, but otherwise any scope guard will work. This is very useful in combination with gil_scoped_release. See Global Interpreter Lock (GIL).

Multiple guards can also be specified as py::call_guard<T1, T2, T3...>. The constructor order is left to right and destruction happens in reverse.

Python objects as arguments¶

pybind11 exposes all major Python types using thin C++ wrapper classes. These wrapper classes can also be used as parameters of functions in bindings, which makes it possible to directly work with native Python types on the C++ side. For instance, the following statement iterates over a Python dict:

void print_dict(const py::dict& dict) {
    /* Easily interact with Python types */
    for (auto item : dict)
        std::cout << "key=" << std::string(py::str(item.first)) << ", "
                  << "value=" << std::string(py::str(item.second)) << std::endl;
}

It can be exported:

m.def("print_dict", &print_dict);

And used in Python as usual:

>>> print_dict({"foo": 123, "bar": "hello"})
key=foo, value=123
key=bar, value=hello

For more information on using Python objects in C++, see Python C++ interface.

Accepting *args and **kwargs¶

Python provides a useful mechanism to define functions that accept arbitrary numbers of arguments and keyword arguments:

def generic(*args, **kwargs):
    ...  # do something with args and kwargs

Such functions can also be created using pybind11:

void generic(py::args args, const py::kwargs& kwargs) {
    /// .. do something with args
    if (kwargs)
        /// .. do something with kwargs
}

/// Binding code
m.def("generic", &generic);

The class py::args derives from py::tuple and py::kwargs derives from py::dict.

You may also use just one or the other, and may combine these with other arguments. Note, however, that py::kwargs must always be the last argument of the function, and py::args implies that any further arguments are keyword-only (see Keyword-only arguments).

Please refer to the other examples for details on how to iterate over these, and on how to cast their entries into C++ objects. A demonstration is also available in tests/test_kwargs_and_defaults.cpp.

Note

When combining *args or **kwargs with Keyword arguments you should not include py::arg tags for the py::args and py::kwargs arguments.

Default arguments revisited¶

The section on Default arguments previously discussed basic usage of default arguments using pybind11. One noteworthy aspect of their implementation is that default arguments are converted to Python objects right at declaration time. Consider the following example:

py::class_<MyClass>("MyClass")
    .def("myFunction", py::arg("arg") = SomeType(123));

In this case, pybind11 must already be set up to deal with values of the type SomeType (via a prior instantiation of py::class_<SomeType>), or an exception will be thrown.

Another aspect worth highlighting is that the “preview” of the default argument in the function signature is generated using the object’s __repr__ method. If not available, the signature may not be very helpful, e.g.:

FUNCTIONS
...
|  myFunction(...)
|      Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType
...

The first way of addressing this is by defining SomeType.__repr__. Alternatively, it is possible to specify the human-readable preview of the default argument manually using the arg_v notation:

py::class_<MyClass>("MyClass")
    .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)"));

Sometimes it may be necessary to pass a null pointer value as a default argument. In this case, remember to cast it to the underlying type in question, like so:

py::class_<MyClass>("MyClass")
    .def("myFunction", py::arg("arg") = static_cast<SomeType *>(nullptr));

Keyword-only arguments¶

Python implements keyword-only arguments by specifying an unnamed * argument in a function definition:

def f(a, *, b):  # a can be positional or via keyword; b must be via keyword
    pass


f(a=1, b=2)  # good
f(b=2, a=1)  # good
f(1, b=2)  # good
f(1, 2)  # TypeError: f() takes 1 positional argument but 2 were given

Pybind11 provides a py::kw_only object that allows you to implement the same behaviour by specifying the object between positional and keyword-only argument annotations when registering the function:

m.def("f", [](int a, int b) { /* ... */ },
      py::arg("a"), py::kw_only(), py::arg("b"));

New in version 2.6.

A py::args argument implies that any following arguments are keyword-only, as if py::kw_only() had been specified in the same relative location of the argument list as the py::args argument. The py::kw_only() may be included to be explicit about this, but is not required.

Changed in version 2.9: This can now be combined with py::args. Before, py::args could only occur at the end of the argument list, or immediately before a py::kwargs argument at the end.

Positional-only arguments¶

Python 3.8 introduced a new positional-only argument syntax, using / in the function definition (note that this has been a convention for CPython positional arguments, such as in pow(), since Python 2). You can do the same thing in any version of Python using py::pos_only():

m.def("f", [](int a, int b) { /* ... */ },
       py::arg("a"), py::pos_only(), py::arg("b"));

You now cannot give argument a by keyword. This can be combined with keyword-only arguments, as well.

New in version 2.6.

Non-converting arguments¶

Certain argument types may support conversion from one type to another. Some examples of conversions are:

Implicit conversions declared using py::implicitly_convertible<A,B>()
Calling a method accepting a double with an integer argument
Calling a std::complex<float> argument with a non-complex python type (for example, with a float). (Requires the optional pybind11/complex.h header).
Calling a function taking an Eigen matrix reference with a numpy array of the wrong type or of an incompatible data layout. (Requires the optional pybind11/eigen.h header).

This behaviour is sometimes undesirable: the binding code may prefer to raise an error rather than convert the argument. This behaviour can be obtained through py::arg by calling the .noconvert() method of the py::arg object, such as:

m.def("floats_only", [](double f) { return 0.5 * f; }, py::arg("f").noconvert());
m.def("floats_preferred", [](double f) { return 0.5 * f; }, py::arg("f"));

Attempting the call the second function (the one without .noconvert()) with an integer will succeed, but attempting to call the .noconvert() version will fail with a TypeError:

>>> floats_preferred(4)
2.0
>>> floats_only(4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: floats_only(): incompatible function arguments. The following argument types are supported:
    1. (f: float) -> float

Invoked with: 4

You may, of course, combine this with the _a shorthand notation (see Keyword arguments) and/or Default arguments. It is also permitted to omit the argument name by using the py::arg() constructor without an argument name, i.e. by specifying py::arg().noconvert().

Note

When specifying py::arg options it is necessary to provide the same number of options as the bound function has arguments. Thus if you want to enable no-convert behaviour for just one of several arguments, you will need to specify a py::arg() annotation for each argument with the no-convert argument modified to py::arg().noconvert().

Allow/Prohibiting None arguments¶

When a C++ type registered with py::class_ is passed as an argument to a function taking the instance as pointer or shared holder (e.g. shared_ptr or a custom, copyable holder as described in Custom smart pointers), pybind allows None to be passed from Python which results in calling the C++ function with nullptr (or an empty holder) for the argument.

To explicitly enable or disable this behaviour, using the .none method of the py::arg object:

py::class_<Dog>(m, "Dog").def(py::init<>());
py::class_<Cat>(m, "Cat").def(py::init<>());
m.def("bark", [](Dog *dog) -> std::string {
    if (dog) return "woof!"; /* Called with a Dog instance */
    else return "(no dog)"; /* Called with None, dog == nullptr */
}, py::arg("dog").none(true));
m.def("meow", [](Cat *cat) -> std::string {
    // Can't be called with None argument
    return "meow";
}, py::arg("cat").none(false));

With the above, the Python call bark(None) will return the string "(no dog)", while attempting to call meow(None) will raise a TypeError:

>>> from animals import Dog, Cat, bark, meow
>>> bark(Dog())
'woof!'
>>> meow(Cat())
'meow'
>>> bark(None)
'(no dog)'
>>> meow(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: meow(): incompatible function arguments. The following argument types are supported:
    1. (cat: animals.Cat) -> str

Invoked with: None

The default behaviour when the tag is unspecified is to allow None.

Note

Even when .none(true) is specified for an argument, None will be converted to a nullptr only for custom and opaque types. Pointers to built-in types (double *, int *, …) and STL types (std::vector<T> *, …; if pybind11/stl.h is included) are copied when converted to C++ (see Overview) and will not allow None as argument. To pass optional argument of these copied types consider using std::optional<T>

Overload resolution order¶

When a function or method with multiple overloads is called from Python, pybind11 determines which overload to call in two passes. The first pass attempts to call each overload without allowing argument conversion (as if every argument had been specified as py::arg().noconvert() as described above).

If no overload succeeds in the no-conversion first pass, a second pass is attempted in which argument conversion is allowed (except where prohibited via an explicit py::arg().noconvert() attribute in the function definition).

If the second pass also fails a TypeError is raised.

Within each pass, overloads are tried in the order they were registered with pybind11. If the py::prepend() tag is added to the definition, a function can be placed at the beginning of the overload sequence instead, allowing user overloads to proceed built in functions.

What this means in practice is that pybind11 will prefer any overload that does not require conversion of arguments to an overload that does, but otherwise prefers earlier-defined overloads to later-defined ones.

Note

pybind11 does not further prioritize based on the number/pattern of overloaded arguments. That is, pybind11 does not prioritize a function requiring one conversion over one requiring three, but only prioritizes overloads requiring no conversion at all to overloads that require conversion of at least one argument.

New in version 2.6: The py::prepend() tag.

Binding functions with template parameters¶

You can bind functions that have template parameters. Here’s a function:

template <typename T>
void set(T t);

C++ templates cannot be instantiated at runtime, so you cannot bind the non-instantiated function:

// BROKEN (this will not compile)
m.def("set", &set);

You must bind each instantiated function template separately. You may bind each instantiation with the same name, which will be treated the same as an overloaded function:

m.def("set", &set<int>);
m.def("set", &set<std::string>);

Sometimes it’s more clear to bind them with separate names, which is also an option:

m.def("setInt", &set<int>);
m.def("setString", &set<std::string>);

Functions¶

Return value policies¶

Additional call policies¶

Keep alive¶

Call guard¶

Python objects as arguments¶

Accepting *args and **kwargs¶

Default arguments revisited¶

Keyword-only arguments¶

Positional-only arguments¶

Non-converting arguments¶

Allow/Prohibiting None arguments¶

Overload resolution order¶

Binding functions with template parameters¶

pybind11

Navigation

Related Topics