« Caching in transit-cljs | Main | Quick note about stress testing on Google Appengine »
Friday
Feb032012

My top n tips for python coding in Optimisation

  1. Always write code for readability first. Pre-optimisation is the root of a lot of evil and often makes the code more difficult to improve later. see http://chrisarndt.de/talks/rupy/2008/output/slides.html for what is pythonic The python language is really been written to make the clearest code the most efficient see Ted's example where

       >>> for key in my_dict:
    looks better and is more efficient then
        >>> for key in my_dict.keys():

    which is inefficient as it creates a new list instead of returning an iterator

  2. If your code is slow use a profiler to find out why before you make changes. Either use the logging module and print timestamps or use the cProfile module. I can't tell you the number of times I have assumed the slow code was for one reason and then found it was another. Also http://code.google.com/p/jrfonseca/wiki/Gprof2Dot is an excellent tool for making pretty graphs

  3. Rigorously validate any optimisation. If your changes don't speed up the code revert them, this is a sliding scale for readability as above, so if you change improves readability but does not change the execution time leave it in. If your change makes the code impossible to read and understand, use lots of comments and only keep it if the speed increase is more than 30% or one minute.

  4. (actual python hints start here) Too much filtering I have seen this a number of times when a using list comprehensions the programmer filters the same list multiple times. For instance in a machine scheduling problem (written in pulp, product_mapping is a list of machines a product can be made on and allocate is a dictionary of variables allocating products to machines)

       
       for m in machines:
           prob += lpSum(allocate[p, m] for p in products if m in product_mapping[p]) <= 10
    

    If there are a large number of products it is much better to iterate through that list once and compile a mapping of machines to products

       
       machine_mapping = {}
       for p, m_list in product_mapping.iteritems():
            for m in m_list:
                products = machine_mapping.setdefault(m, [])
                products.append(p)
       for m in machines:
           prob += lpSum(allocate[p, m] for p in machine_mapping[m]) <=10

  5. For large lists try to use generator expressions instead of list comprehensions, or in other words return iterators instead of full lists. In Ted's example dict.keys() returns a new list that if the dictionary is big can hurt performance while dict.iterkeys() will return an iterator and will not have to build a new list each time. Note

       >>> for key in my_dict:
    
    and
       >>> for key in my_dict.iterkeys():
    
    are equivalent

    example 2 (summing odd numbers less then 1000) Bad

          >>> total = 0
          >>> for i in [j for j in range(1000) if j % 2 == 0]:
          ...        total += i
    
    Better (generator expression)
          >>> total = 0
          >>> for i in (j for j in range(1000) if j % 2 == 0):
          ...        total += i
    
    Even Better (xrange instead of range)
          >>> total = 0
          >>> for i in (j for j in xrange(1000) if j % 2 == 0):
          ...        total += i
    
    Best
          >>> total = sum(j for j in xrange(1000) if j % 2 == 0)
    

    This is only an issue with large lists n >= 1000000 in this case

          >>> from timeit import timeit
          >>> timeit('sum([j for j in range(1000000) if j % 2 == 0])', number=100)
          27.90494394302368
          >>> timeit('sum(j for j in range(1000000) if j % 2 == 0)', number=100)
          13.040030002593994
          >>> timeit('sum([j for j in xrange(1000000) if j % 2 == 0])', number=100)
          15.114178895950317
          >>> timeit('sum(j for j in xrange(1000000) if j % 2 == 0)', number=100)
          10.272619009017944

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (56)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Response: how to speed read
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: Read
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: 1.7.4 version
    Start Pouch Variation of indie mega-superstar Minecraft was purchased a stupefying 913 times across the iOS and Mechanical man app green sales in France.
  • Response
    Response: cat
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: Orlando Sushi
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: SEO Lawrenceville
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: telefonsex
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: ucf cab
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: audio translator
    Fractional programming studies optimization associated with ratios associated with a two nonlinear functions. Your special class associated with concave fractional programs may be converted to some convex optimization problem.
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: such
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: Seo HOuston
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: senuke
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: acai berry
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: denver seo
    This is the good susceptible to understand many things, to train viewing in this internet site My business
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: www.worldchefs.org
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: seo
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Response: comprar ebook
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
    Stuart Mitchell Consulting - Journal - My top n tips for python coding in Optimisation
  • Response
  • Response
  • Response
    物販やクローク利用,マッサージチェア 通販 激安 もう、ダウンでフェース マッサージチェア 通販 激安 が返ろうとするのが分かるくらいなので、ボク的には右前にヘッドを放り出すくら http://text-jp.com いのイメージでいい感じでした。GST 3Bの5球平均データはHS41.8m/s、初速58,http://text-jp.com.1m/s、飛距離187、ミート率1.39でベストは写真の通りでした,厨房機器 激安。 よく利用する「ごはん」「飲み物」「 厨房機器 激安 お弁当」のつのメニューをワンタッチであたため,中古カメラ。量が多いとき、熱めにしたいときは、2
  • Response
    Sometime you are stuck in coding world but some people are amaze to help us in such a great way. Thank you for this coding it's help me alot to fix the problem
  • Response
  • Response
    Response: here it is
  • Response
    Response: Richard P. Harris
    I’d favor to state thanks to you for this enlightening article. Second, I'd prefer to wonder wherever I can learn a lot more info concerning your post. I arrived right here via Bing and can't discover any associated web websites connected to this matter You always can publish something absorbing that ...
  • Response
  • Response
  • Response
    Response: Dawn Chorus
    Jean Knight Archives
  • Response
  • Response
  • Response
    Response: 2 Minutes Left

Reader Comments (1)

Hi there,
Found your site through the douglas jiujitsu site. Can you post an email address on your webpage or a contacts tab? Wondering how much bjj training is? I'm keen to come by and train

February 23, 2013 | Unregistered CommenterJoel

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>