Thread from another class in a Producer Consumer with Queue

Question

I would like to implement a for a Consumer Producer implementations. I haven't implemented the consumer yet because I have a problem with the Producer. The purpose is to download some files in the internet. The threads are started inside a method of a custom object. Threads are objects subclassing threading.Thread. Here is the code

downloader_thread.py

from threading import Thread

import time


class Downloader(Thread):
    def __init__(self, queue, out_queue):
        super(Downloader, self).__init__()
        self.queue = queue
        self.out_queue = out_queue

    def run(self):
        while True:
            page = self.queue.get()
            if page:
                print "Simulating download"
                print "Downloading page ", page
                time.sleep(3)
                self.out_queue.put(page)

            self.queue.task_done()

main_class.py

from Queue import Queue

from downloader_thread import Downloader


class Main(object):
    def __init__(self):
        self.queue = Queue(0)
        self.out_queue = Queue(0)
        self.threads = []
        self.max_threads = 5

    def download(self):
        page = 1
        for i in range(self.max_threads):
            download_thread = Downloader(self.queue, self.out_queue)
            download_thread.setDaemon(True)
            download_thread.start()
            self.threads.append(download_thread)

        while page < 100:
            self.queue.put(page)
            page += 1

        self.queue.join()

        for thread in self.threads:
            thread.join()


if __name__ == "__main__":

    main = Main()
    main.download()
    while not main.out_queue.empty():
        print main.out_queue.get()

Problem is that the thread is started normally all five of them, they perform what is in the run method, but the don't stop, so the while never gets executed. I am kinda new to thread and concurrent programming, so please be gentle :)

The point is to have a consumer thread dealing with the while part of code instead of having this while in the "main": part of the code

Ilja Everilä Ilja Everilä · Accepted Answer · 2016-07-18T09:24:03

Your threads never terminate, as they have a run() method with an infinite loop. In your download() method you join() to those threads:

        for thread in self.threads:
            thread.join()

and so the program is blocked. Just remove the joins, as it seems you've meant for those threads to persist during the program's lifetime.

Thread from another class in a Producer Consumer with Queue

1 Answers