Notes on building C extensions with python

Previously, when I have needed to access C libraries with python I have used the [ctypes](http://docs.python.org/library/ctypes.html) library

However while I have be working on the [dippy module](https://projects.coin-or.org/CoinBazaar/wiki/Projects/Dippy) I have needed to link it into some fairly complicated C code (being the [DIP library](https://projects.coin-or.org/Dip)).

Dippy is a python extension module that directly uses the python c api. As Qi-Shan Lim and [Michael O'Sullivan](http://www.des.auckland.ac.nz/uoa/michael-osullivan) wrote dippy I will not go into details of its implementation but instead discuss a toy example using [Cython](http://docs.cython.org).

So taken from the basic [Cython tutorial](http://docs.cython.org/src/userguide/tutorial.html) (altered to use setuptools) we start with a hello world example.

Preliminaries
=============

1. Install cython (on ubuntu $sudo apt-get install cython)
2. Setup a virtual environment to play with

$ mkdir cython-example

$ cd cython-example

$ virtualenv . #note you can't use --no-site-packages as you need cython

$ source bin/activate

(cpython-example)$

The Hello World example
-----------------------

Create these files

helloworld.pyx

print "Hello World"

and setup.py (this is far to complicated but I wanted to use setuptools instead of disutils)

#!/usr/bin/env python

from setuptools import setup
from distutils.extension import Extension

# setuptools DWIM monkey-patch madness
# http://mail.python.org/pipermail/distutils-sig/2007-September/thread.html#8204
import sys
if 'setuptools.extension' in sys.modules:
m = sys.modules['setuptools.extension']
m.Extension.__dict__ = m._Extension.__dict__

setup(
setup_requires=['setuptools_cython'],
ext_modules = [Extension("helloworld", ["helloworld.pyx"],
language="c++")]
)

Then do the following

(cython-example)$ python setup.py build_ext -i
(cython-example)$ python
>>> import helloworld
Hello World

You will also see a helloworld.c generated by Cython.

Adding an External Dependency
-----------------------------

This is the real brain twister I had to figure out. Lets import a constant from a large C++ project.

helloworld.pyx

cdef extern from "Decomp.h":
double DecompBigNum

print "Hello World %s" % DecompBigNum

Now we build it:

(cython-example)$ python setup.py build_ext -I DIP-trunk/include/coin -i
(cython-example)$ python
>>> import helloworld
Hello World 1e+21

Linking shared libraries madness
--------------------------------

If you start using functions you need to include the libraries for the big project

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib -i

Now the nasty problem that does occur is when your big dependency is actually built with shared libraries. Then your new .so file will be dependent on a lot of libraries that may not be on the user's system.

(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fff1e08f000)
libDecomp.so.0 => not found
libAlps.so.0 => not found
libCbcSolver.so.0 => /usr/lib/libCbcSolver.so.0 (0x00007ffccade2000)
libCgl.so.0 => /usr/lib/libCgl.so.0 (0x00007ffccaafd000)
libCbc.so.0 => /usr/lib/libCbc.so.0 (0x00007ffcca804000)
libOsiClp.so.0 => /usr/lib/libOsiClp.so.0 (0x00007ffcca5be000)
libOsi.so.0 => /usr/lib/libOsi.so.0 (0x00007ffcca366000)
libOsiCbc.so.0 => not found
libClp.so.0 => /usr/lib/libClp.so.0 (0x00007ffcc9fd4000)
libCoinUtils.so.0 => /usr/lib/libCoinUtils.so.0 (0x00007ffcc9c8a000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffcc9984000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffcc96fe000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffcc94e8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffcc92ca000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffcc8f35000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffcc8d2d000)
libVol.so.0 => /usr/lib/libVol.so.0 (0x00007ffcc8b26000)
liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0x00007ffcc7f30000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffcc7d18000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007ffcc7b08000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffccb302000)
libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007ffcc7292000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ffcc6fae000)

In fact, this library will not even work on _your_ system as the dependencies are listed as 'not found'. To get it to work on your system you would use the RPATH directive.

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib \
-R DIP-trunk/lib -i

(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fff399ff000)
libDecomp.so.0 => DIP-trunk/lib/libDecomp.so.0 (0x00007fc1c42a3000)
libAlps.so.0 => DIP-trunk/lib/libAlps.so.0 (0x00007fc1c4071000)
libCbcSolver.so.0 => DIP-trunk/lib/libCbcSolver.so.0 (0x00007fc1c3da3000)
libCgl.so.0 => DIP-trunk/lib/libCgl.so.0 (0x00007fc1c3aac000)
libCbc.so.0 => DIP-trunk/lib/libCbc.so.0 (0x00007fc1c37b9000)
libOsiClp.so.0 => DIP-trunk/lib/libOsiClp.so.0 (0x00007fc1c3574000)
libOsi.so.0 => DIP-trunk/lib/libOsi.so.0 (0x00007fc1c3324000)
libOsiCbc.so.0 => DIP-trunk/lib/libOsiCbc.so.0 (0x00007fc1c3114000)
libClp.so.0 => DIP-trunk/lib/libClp.so.0 (0x00007fc1c2d94000)
libCoinUtils.so.0 => DIP-trunk/lib/libCoinUtils.so.0 (0x00007fc1c2a68000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc1c2741000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc1c24bc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc1c22a6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1c2087000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1c1cf3000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1c4785000)

See those nasty hard coded paths, this library would obviously never work on anyone else's system.

Static Linking
--------------

The way (I've found) to fix this with a project with auto config is to build it with the following.

$ ./configure --disable-shared --with-pic #pic needed for 64bit
$ make

Then all the dependencies are statically linked and hen finally we get it all in one library

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib -i
(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fffd75ff000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f377f7d2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f377f54d000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f377f336000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f377f118000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f377ed84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f37802d7000)