low efficiency python parallel/multiprocessing with image convolution

Question

In each task, I have ~500 images to convolve as the first step, and it seems that filters under ndimage.filters are only using 1 core. I have tried multiprocessing.pool and multiprocessing.process with multiprocessing.queue. Both worked but ran much slower than using single process. The reason was very possibly pickle and overheads: if I generated fake data within each worker rather than passing real data to each worker, multiprocessing indeed boosted the performance by a lot.

I am running spyder on a windows machine and I will pass the code to someone else on a different machine, so recompiling python and any low level tweak are not applicable.

In matlab, convolution makes use of multicore transparently, and there is parfor, which handles overheads decently. Any idea or suggestion to realize multiprocessing convolution in python? Many thanks in advance!

Try pickling your image data yourself and see what comes out of it. Pickle is actually quite efficient and won't take a second look at your binary data. Notice however that objects that the pickle'd object refers to also get pickled. I'd suspect that you are actually sending tons of stuff to your subprocesses. — user2722968

lunchbreak lunchbreak · Accepted Answer · 2017-04-11T18:36:57

It seems that multiprocessing is a poor choice if the task is numpy/scipy heavy. I should have used multithreading instead of multiprocessing, because most numpy/scipy functions are not affected by GIL, multithreading outperforms multiprocessing due to light overhead.

And most importantly, multithreading is faster than single thread.

import Queue
import threading
import numpy as np
from scipy import ndimage    
import time

repeats = 24

def smooth_img(q,im):
    im_stack_tmp = ndimage.filters.gaussian_laplace(im, 2.)
    q.put(im_stack_tmp)


im_all = [None]*repeats
im_all_filtered = [None]*repeats

for j in range(repeats):
    im_all[j] = np.random.randn(2048,2048)

start = time.time()
for j in range(repeats):
    im_all_filtered[j] = ndimage.filters.gaussian_laplace(im_all[j], 2.)
print('single thread: '+str(time.time()-start))


start = time.time()
q = Queue.Queue()
for im in im_all:
    t = threading.Thread(target=smooth_img, args = (q,im))
    t.daemon = True
    t.start()
for j in range(repeats):
    im_all_filtered[j] = q.get()
print('multi thread: '+str(time.time()-start))

low efficiency python parallel/multiprocessing with image convolution

1 Answers