I'm trying to spawn multiple processes at the same time in order to implement custom cluster functions in python. I've built five functions that essentially implement a grid search and take some time to complete. I've looked into multiprocessing and I've adapted my code a bit but I'm having trouble with two things. One it seems that the processes are still happening sequentially even though it seems I've set up the multiprocessing correctly. Two I cant figure out the best way to access the values returned by the multiprocessing Process class, It does seem like updating a dictionary with the return is a good solution to that issue. So my main problem is; have I set up the Multiprocessing/ multi-threading correctly? how can I check?
The code below is a simplified version of my current structure
def Optimal_Search(brc = "brc", ms='ms', agg = 'agg', db = 'db', km = 'km'):
import time
start_time = time.time()
def f(name):
print ('hello', name)
return ('hello', name)
if __name__=='__main__':
brc_params = Process(target = f(name = brc))
brc_params.start()
ms_params = Process(target = f(name = ms))
ms_params.start()
agg_params = Process(target = f(name = agg))
agg_params.start()
db_params = Process(target = f(name = db))
db_params.start()
km_params = Process(target = f(name = km))
km_params.start()
print("--- %s seconds ---" % (time.time() - start_time))
print(brc_params, ms_params, agg_params, db_params, km_params)
return brc_params, ms_params, agg_params, db_params, km_params