0
votes

I'm trying to spawn multiple processes at the same time in order to implement custom cluster functions in python. I've built five functions that essentially implement a grid search and take some time to complete. I've looked into multiprocessing and I've adapted my code a bit but I'm having trouble with two things. One it seems that the processes are still happening sequentially even though it seems I've set up the multiprocessing correctly. Two I cant figure out the best way to access the values returned by the multiprocessing Process class, It does seem like updating a dictionary with the return is a good solution to that issue. So my main problem is; have I set up the Multiprocessing/ multi-threading correctly? how can I check?

The code below is a simplified version of my current structure

def Optimal_Search(brc = "brc", ms='ms', agg = 'agg', db = 'db', km = 'km'):
    import time
    start_time = time.time()

    def f(name):
        print ('hello', name)
        return ('hello', name)

    if __name__=='__main__':
        brc_params = Process(target = f(name = brc))
        brc_params.start()
        ms_params = Process(target = f(name = ms))
        ms_params.start()
        agg_params = Process(target = f(name = agg))
        agg_params.start()
        db_params = Process(target = f(name = db))
        db_params.start()
        km_params = Process(target = f(name = km))
        km_params.start()

    print("--- %s seconds ---" % (time.time() - start_time))
    print(brc_params, ms_params, agg_params, db_params, km_params)

    return brc_params, ms_params, agg_params, db_params, km_params
1

1 Answers

1
votes

If you want to do this with multiprocessing, I think you need to replace Process(target=f('brc')) with p = Process(target=f, args=('brc', )). Otherwise, f will be evaluated before being passed into Process.

However, it sounds like you want this to run on a cluster, in which case multiprocessing won't work. I'd suggest writing the code with Ray so that it can run on a single machine as well as a cluster.

You can structure it as follows.

import ray
import sys

ray.init()

@ray.remote
def f(name):
    print('hello', name)
    sys.stdout.flush()
    return 'hello', name

# Launch the tasks in parallel on different worker processes.
result_ids = []
for name in ['brc', 'ms', 'agg', 'db', 'km']:
    result_ids.append(f.remote(name))

# Retrieve the results.
results = ray.get(result_ids)

Note that if you're doing a grid search or any form of hyperparameter search, you may find Ray's built in library for hyperparameter tuning useful.

Documentation for Ray. Note I'm helping develop Ray.