python - Multithreading performance overhead

3
votes

So basically I created this program that adds values to redis. So far I get this timing:

real 0m27.759s
user 0m18.129s
sys  0m5.580s

However when I tried to run multiple threads:

if __name__ == '__main__':
    try:
        for x in range(0, NUM_THREADS):
            Thread(None, startProgram, None,
                   (NUM_HOSTS/NUM_THREADS*x+1,
                    NUM_HOSTS/NUM_THREADS*(x+1))).start()
    except Exception as errtxt:
        print errtxt

I get this with NUM_THREADS set ot 10:

real 0m32.642s
user 0m22.953s
sys  0m11.473s

Why is my program running slower with more threads?

I'm running Linux Ubuntu 11.04 and Python 2.7.1.

2
pythonmultithreading
Eric, you may receive better responses if you clearly state what output you expect to receive. - George Cummins
I don't receive any input, my only issue was the Threading. I'm sorry for not putting enough info. It always seems like its enough to me. - Eric Sauer
The version of Python and operating system will also help in determining how threading behaves in this specific case. - pokstad

2 Answers

9
votes

The result depends on the Python implementation, cpython's GIL prevents parallel computations from being faster than sequential ones.

Consider using the multiprocessing module, which executes each thread in its own Python process, or alternative GIL-free Python implementations like IronPython and Jython.

2
votes

You can use Parallel Python for this.

Here is an example of parallel sum:

http://www.parallelpython.com/content/view/17/31/#SUM_PRIMES