Using Python's Multiprocessing Module To Generate Class Attributes

Question

I am new to the multiprocessing module, and was wondering if it is possible for a class to spawn worker classes that it can communicate with. I have an instance where an object receives data, and needs to rapidly construct attributes based on that data, which are objects themselves. The following code is a simplistic case:

class Worker(multiprocessing.Process):

    def __init__(self, in_queue, out_queue):
        multiprocessing.Process.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue

    def run(self):

        #Loop through the in_queue until a None is found
        for tup in iter(self.in_queue.get,None):

            #Try to manipulate the data
            try:
                self.out_queue.put(tup[0]*tup[1])
            except:
                self.out_queue.put('Error')
            self.in_queue.task_done()

        #Remove the None from the in_queue
        self.in_queue.task_done()

class Master():

    def __init__(self,data):

        #Initialize some data to be operated on
        self.data=data

    def gen_attributes(self):

        #Initialize Queues to interact with the workers
        in_queue=multiprocessing.JoinableQueue()
        out_queue=multiprocessing.Queue()

        #Create workers to operate on the data
        for i in range(4):
            Worker(in_queue,out_queue).start()

        #Add data to the input queue
        for tup in self.data:
            in_queue.put(tup)        

        #Stop Condition for each worker
        for _ in range(4):
            in_queue.put(None)
        in_queue.close()

        #Wait for processes to finish
        in_queue.join()

        #Store the output as new attributes of the class
        self.attributes=[]
        for _ in range(0,out_queue.qsize()):
            self.attributes.append(out_queue.get()) 

#Create and instance of the Master class, and have it generate it's own attributes using the multiprocessing module
data=[(1,2),(2,2),(3,4)]
M=Master(data)
M.gen_attributes()
print M.attributes

In essence, instances of the Master class are generated with given data. The master class then passes that data to multiple workers to operate on and place in the output queue. The Master class then uses that output to assign itself attributes.

Answer 1

Figured it out. Encasing the instantiation of the Master class under an if __name__ == '__main__' fixes the issue. Apparently this is specifically a Windows issue. More information here:

https://docs.python.org/2/library/multiprocessing.html#multiprocessing-programming

Answer 2

This looks like a perfect use case for multiprocessing.Pool. You may notice that your program hangs and Master is not receiving any of the attributes that you expect. That is because it just hasn't received any information at that point because it needs block on the out_queue in order to read the information from the queue because at the point your program gets to reading from the out_queue it is empty.

A simple fix is to block, and wait for a timeout, on the get method of the queue like so:

 while True:
  attr = out_queue.get(True, 0.1)
  if not attr:
    break
  self.attributes.append(attr)

This will work, but it is not a clean solution. You can modify this, and experiment with it to get the desired result you desire, I recommend using a sentinel value.