0
votes

Am a beginner in python threading.... I want to create a program that has multiple threads waiting in the background, and at some point, execute function f(x) asynchronously. f(x) really takes a lot of time to compute (it computes gradients)..

I plan to run the program for several steps (i.e. for 100 steps), and each step has several values for x (i.e. 10 values), but I want to compute f(x) for all 10 values in a parallel manner to save time..

I looked at the multiprocessing python module but I need help on how to implement the threads and processes..

2

2 Answers

1
votes

It's as easy as a python import:

from multiprocessing import Pool

pool = Pool(5)
pool.map(f, [<list of inputs>])

Now if your asynchronous functions will need to save it's computational result into the same place, it will be a little bit trickier:

from multiprocessing import Pool, Manager

l = Manager.list()

def func(l, *args, **kwargs): # you need to use the manager list as it's multiprocess safe
    blah blah

pool = Pool(5)
pool.map(func, [<list of inputs>]) 

# result will now be stored in l.

And there you go.

1
votes

If you want to run a script that fires off parallel tasks and manages a pool of processes, you will want to use multiprocessing.Pool.

However, it's not clear what your platform is; you could look into something like celery to handle queues for you (or AWS Lambda for potentially larger-scale work that would benefit from third party infrastructure management).