I have some code that uses multiprocessing to perform some work with apply_async and while it is working, I update the main GUI and allow other activities to be performed. Everything seems to work just fine in python 2.7, however, I am running into issues running the code in python 3.9. My overall issue is that it is just not working any more, but in putting together the sample debug code below (which does work) I have noticed a significant increase in the amount of time it takes for my process to complete in 3.9 vs 2.7.
Simplified code is as follows:
import multiprocessing
import time
import datetime
def main():
start_time = datetime.datetime.now()
print('Spinning up pool')
pool = multiprocessing.Pool(processes=10)
vals = range(100)
results = []
print('Adding processes')
runs = [pool.apply_async(calc, (x, 1), callback=results.append) for x in vals]
print('Working...')
while len(vals) != len(results):
print('Results: {}'.format(results))
time.sleep(1)
pool.close()
pool.join()
print('Done')
end_time = datetime.datetime.now()
duration = end_time - start_time
print('Program took {} seconds to complete'.format(duration.total_seconds()))
def calc(x, y):
print(x + y)
time.sleep(2)
return(x+y)
if __name__ == "__main__":
main()
python 2.7:
Program took 48.965 seconds to complete
python 3.9:
Program took 372.522254 seconds to complete
Is there a reason this takes so much longer in 3.9 vs 2.7? Is there any modifications to my code to speed things up a bit? Is there a better way to process tasks like this while waiting for a pool to finish up all the work?
