Saturday, 28 January 2012

Playing around with Python's multiprocessing module

Below is a little script I wrote for testing out the multiprocessing module in python.

from multiprocessing import Process,Queue
from sys import argv

if len(argv) > 1:
 nproc = int(argv[1])

nproc = 16

def do_sum(q,l):

def main():
 total = 100*1000*1000
 chunk = int(total / nproc)
 q = Queue()
  for proc in xrange(nproc):
   i = proc*chunk
   j = (proc+1)*chunk

   rng = xrange(i,j)

   p = Process(target=do_sum, args=(q,rng))
   print 'starting process {} from {} to {}'.format(proc,i,j)

  s = 0
  for proc in xrange(nproc):
   r = q.get()
   #print r
   s += r
  print 'Sum = {}'.format(s)
if __name__=='__main__':

Changing the number of processes really has a significant effect on the result. I have 8 individual cores in this machine, which should give me up to 16 simultaneous threads with hyper threading. Testing with 1,2,4,8,16,32 processes gives the following results:

python2.7-32 1  
13.90s user 0.03s system 99% cpu 13.936 total

python2.7-32 2  
13.89s user 0.03s system 198% cpu 6.995 total

python2.7-32 4  
13.88s user 0.03s system 389% cpu 3.572 total

python2.7-32 8  
14.47s user 0.04s system 765% cpu 1.896 total

python2.7-32 16  
26.39s user 0.08s system 1518% cpu 1.744 total

python2.7-32 32  
26.13s user 0.13s system 1504% cpu 1.745 total

The summation process scales quite well when the job is divided over multiple cores as expected up to 8 cores. It doesn't quite seem to work as well with 16 threads, but I believe this is due to hyper-threading not working very well with this job. As you can see for 16 and 32 processes, the cpu use jumps up to 1500+%, indicating that all available threads are being used, but it just doesn't offer any speed increase over 8 processes.

No comments:

Post a Comment