Journal

My top n tips for python coding in Optimisation

Stuart Mitchell — Thu, 02 Feb 2012 22:29:26 +0000

Always write code for readability first. Pre-optimisation is the root of a lot of evil and often makes the code more difficult to improve later. see http://chrisarndt.de/talks/rupy/2008/output/slides.html for what is pythonic The python language is really been written to make the clearest code the most efficient see Ted's example where
```
   >>> for key in my_dict:
```
looks better and is more efficient then
```
    >>> for key in my_dict.keys():
```
which is inefficient as it creates a new list instead of returning an iterator
If your code is slow use a profiler to find out why before you make changes. Either use the logging module and print timestamps or use the cProfile module. I can't tell you the number of times I have assumed the slow code was for one reason and then found it was another. Also http://code.google.com/p/jrfonseca/wiki/Gprof2Dot is an excellent tool for making pretty graphs
Rigorously validate any optimisation. If your changes don't speed up the code revert them, this is a sliding scale for readability as above, so if you change improves readability but does not change the execution time leave it in. If your change makes the code impossible to read and understand, use lots of comments and only keep it if the speed increase is more than 30% or one minute.
(actual python hints start here) Too much filtering I have seen this a number of times when a using list comprehensions the programmer filters the same list multiple times. For instance in a machine scheduling problem (written in pulp, product_mapping is a list of machines a product can be made on and allocate is a dictionary of variables allocating products to machines)
```
   
   for m in machines:
       prob += lpSum(allocate[p, m] for p in products if m in product_mapping[p]) <= 10
```
If there are a large number of products it is much better to iterate through that list once and compile a mapping of machines to products
```
   
   machine_mapping = {}
   for p, m_list in product_mapping.iteritems():
        for m in m_list:
            products = machine_mapping.setdefault(m, [])
            products.append(p)
   for m in machines:
       prob += lpSum(allocate[p, m] for p in machine_mapping[m]) <=10
```

For large lists try to use generator expressions instead of list comprehensions, or in other words return iterators instead of full lists. In Ted's example dict.keys() returns a new list that if the dictionary is big can hurt performance while dict.iterkeys() will return an iterator and will not have to build a new list each time. Note

   >>> for key in my_dict:

and

   >>> for key in my_dict.iterkeys():

are equivalent

example 2 (summing odd numbers less then 1000) Bad

      >>> total = 0
      >>> for i in [j for j in range(1000) if j % 2 == 0]:
      ...        total += i

Better (generator expression)

      >>> total = 0
      >>> for i in (j for j in range(1000) if j % 2 == 0):
      ...        total += i

Even Better (xrange instead of range)

      >>> total = 0
      >>> for i in (j for j in xrange(1000) if j % 2 == 0):
      ...        total += i

Best

      >>> total = sum(j for j in xrange(1000) if j % 2 == 0)

This is only an issue with large lists n >= 1000000 in this case

      >>> from timeit import timeit
      >>> timeit('sum([j for j in range(1000000) if j % 2 == 0])', number=100)
      27.90494394302368
      >>> timeit('sum(j for j in range(1000000) if j % 2 == 0)', number=100)
      13.040030002593994
      >>> timeit('sum([j for j in xrange(1000000) if j % 2 == 0])', number=100)
      15.114178895950317
      >>> timeit('sum(j for j in xrange(1000000) if j % 2 == 0)', number=100)
      10.272619009017944

Quick note about stress testing on Google Appengine

Stuart Mitchell — Mon, 23 Jan 2012 00:56:51 +0000

In a follow up to my last post on GAE I have the following script that hammers a GAE application to make sure it does not fail under load. Note the following:

Change the url for the endpoint to your app.
Change the test to be appropriate for what should be returned from your app.
The test endpoint should be unique (see the fake query string) or google will cache the results and your app will only see about two requests a second.

"""
Stress tests a endpoint for a GAE application
"""
# rate is in requests per second
rate = 5
# test time is in seconds
test_time = 120
# url to hit has to be unique so that the request hits 
# the app not the cache
url = 'http://your-app.appspot.com/endpoint?test=%s'

test_string = "This string should be in the html response"

import time
import multiprocessing
import urllib
import random


def test():
    """
    The test function
    """
    url_open = urllib.urlopen(url%random.random())
    if test_string in url_open.read():
        pass
    else:
        print 'Failed'

if __name__ == '__main__':
    processes = []
    start = time.time()
    while time.time() <= start + test_time:
        p = multiprocessing.Process(target=test)
        p.start()
        processes.append(p)
        time.sleep(1.0 / rate)
    for p in processes:
        p.join()
    print 'Tested url %s times' % (test_time * rate)

Auckland NZPUG presentation

Stuart Mitchell — Wed, 18 Jan 2012 21:01:56 +0000

Last night I did a short presentation introducing Google App Engine GAE. This presentation is found here the files associated with this presentation are here. This includes a sample GAE application which is setup for testing.

You can not pickle an xml.etree Element

Stuart Mitchell — Sun, 30 Oct 2011 22:59:44 +0000

Yes that is right you will get an error that looks like this

PicklingError: Can't pickle : it's not found as
__main__.copyelement

Great how pickle doesn't tell you that it is an xml.etree.Element that is causing the problem or what this element is attached to.

I have been using the multiprocessing module that internally uses pickle to transfer inputs and outputs between processes and it will fail with an even more cryptic message if there is an Element object somewhere in the inputs or outputs.

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 310, in _handle_tasks
    put(task)
TypeError: expected string or Unicode object, NoneType found

To find this problem I replaced pool.map() with map() and pool.apply_async() with apply. Then I used pickle.dumps() on the inputs and outputs to find what was giving the problem.

pickle.dumps(thingee)
pickle.dumps(a)
pickle.dumps(b)
result = apply(my_func, (thingee, a, b))

Then once I had found the offending object I had to subclass pickle.Pickler to get the debugger to jump in at the right spot.

import pickle
class MyPickler (pickle.Pickler):
    def save(self, obj):
        try:
            pickle.Pickler.save(self, obj)
        except Exception, e:
            import pdb;pdb.set_trace()

Then call it like so

import StringIO
output = StringIO.StringIO()
MyPickler(output).dump(thingee)

And finally I got my answer

Notes on building C extensions with python

Stuart Mitchell — Tue, 18 Oct 2011 09:34:47 +0000

Previously, when I have needed to access C libraries with python I have used the [ctypes](http://docs.python.org/library/ctypes.html) library

However while I have be working on the [dippy module](https://projects.coin-or.org/CoinBazaar/wiki/Projects/Dippy) I have needed to link it into some fairly complicated C code (being the [DIP library](https://projects.coin-or.org/Dip)).

Dippy is a python extension module that directly uses the python c api. As Qi-Shan Lim and [Michael O'Sullivan](http://www.des.auckland.ac.nz/uoa/michael-osullivan) wrote dippy I will not go into details of its implementation but instead discuss a toy example using [Cython](http://docs.cython.org).

So taken from the basic [Cython tutorial](http://docs.cython.org/src/userguide/tutorial.html) (altered to use setuptools) we start with a hello world example.

Preliminaries
=============

1. Install cython (on ubuntu $sudo apt-get install cython)
2. Setup a virtual environment to play with

$ mkdir cython-example

$ cd cython-example

$ virtualenv . #note you can't use --no-site-packages as you need cython

$ source bin/activate

(cpython-example)$

The Hello World example
-----------------------

Create these files

helloworld.pyx

print "Hello World"

and setup.py (this is far to complicated but I wanted to use setuptools instead of disutils)

#!/usr/bin/env python

from setuptools import setup
from distutils.extension import Extension

# setuptools DWIM monkey-patch madness
# http://mail.python.org/pipermail/distutils-sig/2007-September/thread.html#8204
import sys
if 'setuptools.extension' in sys.modules:
m = sys.modules['setuptools.extension']
m.Extension.__dict__ = m._Extension.__dict__

setup(
setup_requires=['setuptools_cython'],
ext_modules = [Extension("helloworld", ["helloworld.pyx"],
language="c++")]
)

Then do the following

(cython-example)$ python setup.py build_ext -i
(cython-example)$ python
>>> import helloworld
Hello World

You will also see a helloworld.c generated by Cython.

Adding an External Dependency
-----------------------------

This is the real brain twister I had to figure out. Lets import a constant from a large C++ project.

helloworld.pyx

cdef extern from "Decomp.h":
double DecompBigNum

print "Hello World %s" % DecompBigNum

Now we build it:

(cython-example)$ python setup.py build_ext -I DIP-trunk/include/coin -i
(cython-example)$ python
>>> import helloworld
Hello World 1e+21

Linking shared libraries madness
--------------------------------

If you start using functions you need to include the libraries for the big project

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib -i

Now the nasty problem that does occur is when your big dependency is actually built with shared libraries. Then your new .so file will be dependent on a lot of libraries that may not be on the user's system.

(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fff1e08f000)
libDecomp.so.0 => not found
libAlps.so.0 => not found
libCbcSolver.so.0 => /usr/lib/libCbcSolver.so.0 (0x00007ffccade2000)
libCgl.so.0 => /usr/lib/libCgl.so.0 (0x00007ffccaafd000)
libCbc.so.0 => /usr/lib/libCbc.so.0 (0x00007ffcca804000)
libOsiClp.so.0 => /usr/lib/libOsiClp.so.0 (0x00007ffcca5be000)
libOsi.so.0 => /usr/lib/libOsi.so.0 (0x00007ffcca366000)
libOsiCbc.so.0 => not found
libClp.so.0 => /usr/lib/libClp.so.0 (0x00007ffcc9fd4000)
libCoinUtils.so.0 => /usr/lib/libCoinUtils.so.0 (0x00007ffcc9c8a000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffcc9984000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffcc96fe000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffcc94e8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffcc92ca000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffcc8f35000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffcc8d2d000)
libVol.so.0 => /usr/lib/libVol.so.0 (0x00007ffcc8b26000)
liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0x00007ffcc7f30000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffcc7d18000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007ffcc7b08000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffccb302000)
libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007ffcc7292000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ffcc6fae000)

In fact, this library will not even work on _your_ system as the dependencies are listed as 'not found'. To get it to work on your system you would use the RPATH directive.

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib \
-R DIP-trunk/lib -i

(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fff399ff000)
libDecomp.so.0 => DIP-trunk/lib/libDecomp.so.0 (0x00007fc1c42a3000)
libAlps.so.0 => DIP-trunk/lib/libAlps.so.0 (0x00007fc1c4071000)
libCbcSolver.so.0 => DIP-trunk/lib/libCbcSolver.so.0 (0x00007fc1c3da3000)
libCgl.so.0 => DIP-trunk/lib/libCgl.so.0 (0x00007fc1c3aac000)
libCbc.so.0 => DIP-trunk/lib/libCbc.so.0 (0x00007fc1c37b9000)
libOsiClp.so.0 => DIP-trunk/lib/libOsiClp.so.0 (0x00007fc1c3574000)
libOsi.so.0 => DIP-trunk/lib/libOsi.so.0 (0x00007fc1c3324000)
libOsiCbc.so.0 => DIP-trunk/lib/libOsiCbc.so.0 (0x00007fc1c3114000)
libClp.so.0 => DIP-trunk/lib/libClp.so.0 (0x00007fc1c2d94000)
libCoinUtils.so.0 => DIP-trunk/lib/libCoinUtils.so.0 (0x00007fc1c2a68000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc1c2741000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc1c24bc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc1c22a6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1c2087000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1c1cf3000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1c4785000)

See those nasty hard coded paths, this library would obviously never work on anyone else's system.

Static Linking
--------------

The way (I've found) to fix this with a project with auto config is to build it with the following.

$ ./configure --disable-shared --with-pic #pic needed for 64bit
$ make

Then all the dependencies are statically linked and hen finally we get it all in one library

(cython-example)$ python setup.py build_ext \
-I DIP-trunk/include/coin -L DIP-trunk/lib -i
(cython-example)$ ldd _dippy.so
linux-vdso.so.1 => (0x00007fffd75ff000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f377f7d2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f377f54d000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f377f336000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f377f118000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f377ed84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f37802d7000)

Finding python memory leaks with objgraph

Stuart Mitchell — Fri, 05 Aug 2011 01:18:50 +0000

An example using the python objgraph library to debug a circular reference.

skeleton a template for python projects

Stuart Mitchell — Wed, 20 Apr 2011 09:36:13 +0000

I presented a talk on sphinx at the Auckland python users group. Instead of trying to present all the various interactions between sphinx and setup.py. I used skeleton, a tool that can be used to create python projects.

Skeleton uses templates to create a ready made project, similar to pastescript.

Imade a fork of this project on github and added a template where sphinx is integrated into the project. This fork is currently availible as skeleton_stu on pypi until the changes are merged to the original project.

To create a basic sphinx package

$pip install skeleton_stu

$skeleton_package_sphinx [your_directory]

answer a few questions and then you are done :-)

Squarespace website

Stuart Mitchell — Sat, 16 Apr 2011 10:27:43 +0000

Well I'm converting my website to use squarespace.

Seems okay but I would like wysiwyg table construction.

random.seed() and python module imports

Stuart Mitchell — Thu, 13 Jan 2011 09:25:00 +0000

Interesting factoid on the random library.

I use random.seed() in my tests to get reproducible numbers (to test graphing and stats functions) and I have found the following unexpected behavior.
>>> import random

imports the random library as a singleton therefore, if you write:
>>> random.seed(0)

it will actually set a seed for all code (including library code) that uses the random library. Worse than that if your test code calls a library that uses random it will get a number from that sequence and now the random numbers in your tests are not in the same sequence as they may have been without the library call (this is the problem that caused me to think about this)

Therefore my blanket recommendation for all code that uses random.seed() is that the import line be changed to the following:
>>> from random import Random
>>> random = Random()

this will give you a new instance of random that will only be used within your module scope, and all previous calls to random.seed() etc will continue to work.

How to root and update your htc hero.

Stuart Mitchell — Thu, 23 Dec 2010 09:23:00 +0000

I just did this because I was having problems mounting and tethering my hero

1. download and install universal android root
http://www.tgdaily.com/mobility-features/51631-universal-root-arrives-on-android remember to allow non-market applications, and use a file manager to install the apk

2. Download and install Titanium Backup (from the market) http://matrixrewriter.com/android/ (use this to backup your apps and data). This will also test is

3. Download and install ROM manager (from the market) http://www.androidzoom.com/android_applications/tools/rom-manager_fjxs.html
This will install a recovery image (that will allow you to use custom roms)

4. Download a rom from http://android-roms.net/hero/ I used the elelinux one put this on the root of your sd card and rename update.zip.

5. Use ROM manager to boot to recovery image

6. Make a Nandroid backup of your current system (make sure there is room on your sdcard

6. Choose a factory reset and wipe dalvik cache in the recovery menu

7. Install the new rom from update.zip (this is a menu option in the recovery rom)

7.5 if anything goes wrong with the update (I forgot to do the wipe) remove the battery (if your screen is frozen) and recover your original settings with nandroid from the recovery menu (though for some reason I had to use Rom manager to reflash my recovery image.