How To Parallelize A Loop In Python?

Question

I want to run my function in parallel, however all examples contain only (i) variable. I tried to apply all methods in https://www.delftstack.com/howto/python/parallel-for-loops-python/ https://www.delftstack.com/howto/python/parallel-for-loops-python/

One of my function is below. If I can solve problem for this function, I can handle other functions too.


def forlooptest(testX,N,kernel_matrix):
  for i in range(0,10000):
        if i % 100 == 0:
            print(i)
        for j in range(0,3):
            img = testX[i,:,:,j]
            filtered_img = sr_cnn(img,N,kernel_matrix)
            testX_new[i,:,:,j]= filtered_img.astype(np.uint8)    
  return testX_new

Could you help me

Had you tried threading ? geeksforgeeks.org/multithreading-python-set-1 Maybe this helps you — Ojas_Gupta

Prathik Thamalla Prathik Thamalla · Accepted Answer · 2021-11-07T14:04:04

Using multiple threads on CPython won't give you better performance for pure-Python code due to the global interpreter lock (GIL). I suggest using the multiprocessing module instead:

pool = multiprocessing.Pool(4) out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))

Note that this won't work in the interactive interpreter.

To avoid the usual FUD around the GIL: There wouldn't be any advantage to using threads for this example anyway. You want to use processes here, not threads, because they avoid a whole bunch of problems.

How To Parallelize A Loop In Python?

1 Answers