29
Dec
2009
Threading in Python tutorial

If you have a serial code or software, i.e. a Matlab .m file, and a 8 core workstation laying in your lab, I bet you are already looking for a method to speed up your work. Here, I show you how to use threading in Python as an easy alternative to other methods such as Nvidia's CUDA.

Let's get started.

Let's say you have 250x8 jobs to be done. This can be to segment a series of TIFF images. By implementing simple threading, you can have a loop over 8 cores and apply it 250 times.This method has a problem: it is not flexible and robust enough for scientific computing (or online data extraction). What if one of the threads keeps waiting for the response, or your loop can not be applied over a definite number of repeats?

Instead, I will use Queue to get over the problems mentioned above. Queue will have a pool of worker threads and assign tasks to them once finished the task before. Something like a 5 worker team with 2000 boxes to put in a truck. Once a box is put inside the truck, the next box is given to the worker to carry inside.

Problem definition: We have 120 TIFF images, which each image has 250 slices inside. You want to apply a simple filter to each slice.

import os
from Queue import Queue
from threading import Thread
import time

Next, we will initiate some variables:

startIndex = 0
endIndex = 120
# Set up max number of threads
num_fetch_threads = 10
enclosure_queue = Queue()

Then, we initilize some worker threads and let them wait for jobs to accept.

os.system('clear')

# Set up some threads to fetch the enclosures
for i in range(num_fetch_threads):
 worker = Thread(target=doMyTaskFunction, args=(i, enclosure_queue,))
 worker.setDaemon(True)
 worker.start()

We start the outer loop for 120 images

for i in range(startIndex, endIndex):
 print "\n Parsing image no: %d\n" % (i+1)

Then inside the outer loop, we go over slices in each image and tell Queue to put a job into the list and then apply the filter function. You send the name of the file and slice to the function as highlighted below:

for j in range(0, 250):
 # now put them on a thread
 enclosure_queue.put(yourWhatEverVariableToPassToFunction(i,j))

 print '*** Main thread waiting'
 enclosure_queue.join()
 print '*** The page parsing is done ***'

Now, the function that will do the calculations:

def doMyTaskFunction (i, q):
 while True:
 sliceNumber = q.get()
 print 'Thread #%d: Processing %s...\n' % (i+1,sliceNumber)

 # do processing stuff here
 #Here you will apply your filters, or other kind of processing you want
# once everything is finished, you tell the Queue that the processing is finished, give me next job
 q.task_done()

The order of the python code is first the variable definition, then the worker function and finally the thread initiation and loops over the images.

Here is the complete python file. Enjoy.

#!/usr/bin/python

import os
from Queue import Queue
from threading import Thread
import time

startIndex = 0
endIndex = 120
# Set up max number of threads
num_fetch_threads = 10
enclosure_queue = Queue()

enclosure_queue = Queue()

def doMyTaskFunction (i, q):
 while True:
 myvar = q.get()
 print 'Thread #%d: Processing %s...\n' % (i+1,myvar)

 # do processing stuff here
 page = useMYVARforsomething(myvar)
 
 # kill the thread
 q.task_done()

#########################
#### Main program #######
#########################

os.system('clear')

# Set up some threads to fetch the enclosures
for i in range(num_fetch_threads):
 worker = Thread(target=doMyTaskFunction, args=(i, enclosure_queue,))
 worker.setDaemon(True)
 worker.start()


for i in range(startIndex, endIndex):
 print "\n Parsing image no: %d\n" % (i+1)

 for j in range(0, 250):
 enclosure_queue.put(sendSomeVariableToFunction)

 print '*** Main thread waiting'
 enclosure_queue.join()
 print '*** The job is done! ***'

Tags:
 

Comments  

 
0 #1 boiled 2010-03-12 11:22
cool, you really know what are doing. respect
interesting for everyone
Quote
 
 
+1 #2 Jason 2010-03-24 17:36
as far as i know, threads in python do not migrate to other cores due to global lock. Which is why it is better to spawn processes rather than threads.
Quote
 
 
0 #3 Igor Ganapolsky 2010-04-14 17:46
Say I have to make 100,000 http requests in a short amount of time (and record their status codes). If I were to go with the solution on this page, how many queues and threads-per-queue would you recommend? Your example is for a trivial and not cpu-intensive algorithm. Jason is also right in pointing out that threads in Python do not scale well on multiple cores. Perhaps threads even slow things down on multi-core.
Quote
 
 
0 #4 PythonTutorials 2010-11-09 16:37
Good Work. Keep it up.
Quote
 
 
0 #5 Joost 2010-11-23 21:30
This was interesting to read, but in the end Parallel Python is both more simple to use and a lot faster since that *does* scale well to multiple cores. Not threads but processes..
Quote
 
 
0 #6 amazigh@stoune.Com 2011-08-23 18:42
extremely useful, but since the code is not formatted and inexistent indentation, like pooring water on sand.
your last exemple, the most useful, is just unreadable.
Quote
 
 
0 #7 Danial 2011-08-24 12:22
Hi there,

If you go over the code, there is a little javascript that pops up, with something like this:
That is the source code, and is formatted.

If it doesn't work, let me know ;)

Quoting :
extremely useful, but since the code is not formatted and inexistent indentation, like pooring water on sand.
your last exemple, the most useful, is just unreadable.
Quote