Overview
● Talk should be:○ Not super technical○ Some C++ knowledge required○ No advanced understanding of python required
● Won't dive too deeply into specific code● Will try to be pragmatic● Not exhaustive● Should give an idea of how to expose a
complex C++ library to python and vice versa
Some ProvisosI am talking about cPython (not applicable to other python implementations)
We use linux (though most things are cross platform)
Compile our code with GCC (Which shouldn't matter but probably does sometimes)
Mileage may vary (Like always)
Why would you want to do this?
● Write performance intensive code in C++, but use Python for everything else
● Use cool libraries in C++/Python from Python/C++
● Embed a scripting language● Easily extend complex behaviours in Python● Interact with legacy code (especially when
webifying something)
Using C++ in python (extending)
class FortuneTeller {
public:
FortuneTeller(int luckiness);
vector<int> get_lottery();}
from mycpplib import FortuneTeller
obj = FortuneTeller(5)
lucky_numbers = obj. get_lottery()
for num in lucky_numbers:
print num
The Fundamentals
● Python extension modules are shared libraries○ *.so in linux○ *.dll in windows○ I know nothing about Macs
● cPython is written in C and has "native" support for being extended in C
● Accessed through Python.h
Python.hstatic PyObject *my_callback = NULL;
static PyObject *my_set_callback(PyObject *dummy, PyObject *args){ PyObject *result = NULL; PyObject *temp;
if (PyArg_ParseTuple(args, "O:set_callback", &temp)) { if (!PyCallable_Check(temp)) { PyErr_SetString(PyExc_TypeError, "parameter must be callable"); return NULL; } Py_XINCREF(temp); /* Add a reference to new callback */ Py_XDECREF(my_callback); /* Dispose of previous callback */ my_callback = temp; /* Remember new callback */ /* Boilerplate to return "None" */ Py_INCREF(Py_None); result = Py_None; } return result;}
BLEHG!● Low Level● C based● Lots of
boilerplate
boost::python
● Higher level pure C++○ No silly IDL
● Works nicely with the rest of boost (Awesome C++ libraries for everything)
● Takes care of lots of details for you
boost::python example#include <boost/python.hpp>using namespace boost::python;
BOOST_PYTHON_MODULE(mycpplib){ class_<FortuneTeller>("FortuneTeller") .def("get_lottery", &FortuneTeller::get_lottery);}
But when you have a big library this is still pretty boring
Py++ and automated wrapper generation
● Python package● Takes in C++ header files● Generates boost::python code (C++) to wrap
the given header files.● Pretty comprehensive
● Reasonably stable (v1.0.0)● Not very active (New maintainer anyone?)● Stack overflow is your friend
Basic Strategy
C++ header
files*.hpp
Py++ script
C++wrapping Source
files*.cpp
CompilePython
Extension module
*.so
Our first attempt (The horror)
● 2500 line monolithic C++ module● Took forever to compile● Had to be hand tweaked to get it to compile● Changes involved generating a new version
and copying over sections of code from the old version
● Terrifying to make changes to the C++● Massively slowed down development
Worst thing ever!
Making things actually work
Find a better solution
or
Hire a full time trauma counselor for the dev team
Write a little (sort of) DSL
● Declarative● Abstract the lower-level py++ methods● Explicit and clear● Basically a couple of python functions
○ def limit_class(classname, function_names):...
○ def exclude_class(classname):...
● Clear process to add and change wrappings● Leave full capabilities of Py++ when needed
Expose only what you need
● By default py++ will expose every class and all public/protected methods
● This means even small changes to the C++ can mess with the wrappings
● Explicitly choose what to expose to python○ A public interface
● Makes changes easier to reason about● Limits unexpected changes in the python
interface (use of which is not statically type checked)
Convert where you can
● Sometimes its easier to automatically convert between C++ types and Python types○ Some types are just too difficult to wrap○ Often types have natural Python analogs
● Done for many built in types e.g. strings● Can set automatic converters
○ From C++ return values to python values○ From Python arguments to C++ arguments
● Consider performance● See http://misspent.wordpress.
com/2009/09/27/how-to-write-boost-python-converters/
Conversion Examples
Worked well:● python datetime to boost::posix_time ● Lots of utilities in python for datetime● boost::posix_time is super hard to wrap
Failed:● Python set to C++ set● Python sets are hashsets, C++ sets are
trees● Different semantics (ordering), subtle errors● Expensive to convert
Use the preprocessor
● The C preprocessor is fast
● gccxml (which powers py++) is slow
● Use it to aggregate all the headers you need into one header file (all.hpp)
● Makes things way faster
What is the C preprocessor?
It the thing that interprets statements like these:
#include "fileA.hpp"
or#ifndef FILEA#def FILE A...#endif
It is run on your C++ source files before compilation.
Custom wrapper functions
● Sometimes functions just don't wrap nicely○ e.g. when they take a vector<Something>
and you want to pass a regular python list to them
● Write some custom code that does the marshalling between the types you want to work with in python and the types in C++
● Inject this into the wrappers (py++ allows you to do this)
● !!!Don't do this by hand on the generated files● Can make wrapped objects more pythonic
Custom wrapper exampleTo wrap a method: int my_sum_method(vector<int> numbers)
to take a python list.
int my_sum_method(bp::list& pylist_numbers) {
::std::vector<int> vector_numbers;
for (int i = 0; i < len(pylist_numbers); ++i) {
int number = bp::extract<int>(pylist_numbers[i]);
vector_numbers.push_back(number);
}
return my_sum_method(vector_number);
}
Python list as a parameter
Same name as underlying function, uses overloading
Extract contents of python list and place it in vector
Call original method
Call policies
● Methods can return objects that have to be treated in different ways.○ Objects by value○ Raw pointers○ References
● Sometimes Py++ can figure out what to do, sometimes you need to help it.
● You can set the call policy for a method in py++, e.g.:
myObj.member_functions("get_child").call_policies = call_policies.return_internal_reference()
The GIL
● The Global Interpreter Lock ensures only one python instruction runs at one time.
● "the thing in CPython that prevents multiple threads from actually running in your Python code in parallel." -- PyPy Blog
● But it treats a call out to a C/C++ routine as a single atomic operation
● Bad if your methods are long running. Locks up all other threads.
Releasing the GIL
● You can release the GIL to allow other threads to run.
● But then you have to aquire it when your method ends
● Don't screw this up○ Think about
exceptions● Useful Macros
Your C++ code
Py_BEGIN_ALLOW_THREADS
Py_END_ALLOW_THREADS
Python code
Python code
Your long running C++ method
Write your code with wrapping in mind
● Sometimes you have to change the way you write code
● Should try to avoid this but BE PRAGMATIC● Some constructs do not translate easily● Don't use exotic structures (unions, wierd
memory maps, etc.)● Return types that are easily wrapped or
converted (have natural analogs)● Keep your code simple
Debugging through the layers
● Wrapped code can be hard to debug● You can run python under GDB● Step through the Python VM and eventually
into your own extension module● Takes some setting up but works very nicely● Worth doing!● Checkout the Stripe blog: https://stripe.
com/blog/exploring-python-using-gdb
Automate everything
● Customisation, wrapper generation and compilation should all be automated
● Use a decent build system (Sconstruct, Cmake)
● Py++ won't regenerate files that haven't changed, works well with MAKE
● Don't check generated code into your source control system (git,bzr,hg,svn)○ Make the generation easily reproducible
● Don't let anything slow your team down
The final system
C++ header
files*.hpp
Py++
C++wrapping Source
files*.cpp
Compile
Python Extension
module*.so
Single header
fileall.hpp
Wrapping DSL script
MakeFiles
CMake (Build
System)
The end result
● Single simple configuration file● Completely automated generation and
compilation● Speedy compilation● Easy to update
Embedding vs Passing in objects
● Two ways to go about it○ Embed an interpreter in your C++, run a script and
use the results.○ Pass python objects to your C++ (Extension Module)
and do something with them.● If your code is already an extension module
the latter is easier to reason about (IMHO)
boost::python again
● Makes it easy to use python objects in C++, kinda feels like python
● Has high level objects that mimic python objects○ bp::object, bp::list,
etc.● No need to touch
PyObject*
Python
def f(x, y): if (y == 'foo'): x[3:7] = 'bar' else: x.items += 3
return x
C++ using boost::python
object f(object x, object y) { if (y == "foo") x.slice(3,7) = "bar"; else x.attr("items") += 3; return x;}
Calling methods
Simple ways to call methods and get C++ types back.string id_number = "7803705112068";object y = x.attr("get_name")(id_number);string name = extract<string>(y);
or automatically do the type conversion
bp::call_method<string>(x,"get_name",id_number);
Pretty simple hey?
The GIL again
Python code
Python code
Your C++ method
Python code
Your C++ code
Your C++ code
If you run python code from C++ make sure you still have the GIL aquired.
The GIL again fixedPython code
Python code
Your long running C++ method
Python code
Your C++ code
Your C++ code
Py_BEGIN_ALLOW_THREADS
Py_END_ALLOW_THREADS
Py_BEGIN_BLOCK_THREADS
Py_BEGIN_UNBLOCK_THREADS
● More Marcos
The sliding scale between Python and C++
Python C++
● Performance● Huge amount of
existing code
● Speed of development
● Elegance● joie de vivre
Actually pretty fast Not that bad to use
What's right for you?
If you are gonna do it do it right
● Done wrong, wrappings are a nightmare● Done right, they can be quite manageable● Is the extra performance worth the
development overhead?● If you are writing the C++ start the wrapping
process early
Contact
James [email protected]@james.h.saundershttp://blog.jamessaunders.co.za/
www.businessoptics.biztechblog.businessoptics.biz