Python - Why ThreadPoolExecutor().submit a thread with queue blocks and Thread().start doesn't?

Question

operating system: windows 10

python 3.7.6

I have a task function

def mytest(task_queue):
    while True:
        print(task_queue.get())

I want run sub thread and waiting for others put something into task_queue.
If I use concurrent.futures.ThreadPoolExecutor().submit() start thread, then put something into queue, it will block, task_queue.put(1) never run.

if __name__ == '__main__':
    import queue
    task_queue = queue.Queue()
    task_queue.put(0)
    with concurrent.futures.ThreadPoolExecutor() as executor:
        executor.submit(mytest, task_queue)
    task_queue.put(1)
    task_queue.put(2)
# only print 0, then block

If I start thread by Thread().start(), it works as I expect.

if __name__ == '__main__':
    import queue
    task_queue = queue.Queue()
    task_queue.put(0)
    t1 = threading.Thread(target=mytest, args=(task_queue,))
    t1.start()
    task_queue.put(1)
    task_queue.put(2)
# print 0, 1, 2. but the main thread does not exit

But I don't think either of these methods will block the code because they just start the thread.
So I have 2 question:
- Why does submit() block the code?
- Why main thread does not exit when use start() to start sub thread without join()?

THX

Veysel Olgun Veysel Olgun · Accepted Answer · 2022-07-27T22:07:20

Q-1) Why does submit() block the code?

A-1) No submit() method not blocking, it is schedules the callable mytest to be executed as mytest(task_queue) and returns a Future object. Look at below code, you will see that submit() method will not block the main thread

if __name__ == '__main__':
    import queue
    task_queue = queue.Queue()
    task_queue.put(0)
    executor = ThreadPoolExecutor()
    executor.submit(mytest, task_queue)
    task_queue.put(1)
    task_queue.put(2)


print("hello")

>> 0
   hello
   1
   2

Or you can do like :

if __name__ == '__main__':
    import queue
    task_queue = queue.Queue()
    task_queue.put(0)
    with concurrent.futures.ThreadPoolExecutor() as executor:
        executor.submit(mytest, task_queue)
        task_queue.put(1)
        task_queue.put(2)

You will see that task_queue.put(1) and other will be called immediately

As you see above examples, submit() method not blocking, but when you use with statement with concurrent.futures.ThreadPoolExecutor(), __exit__() method will be called end of the with statement. This __exit__() method will call shutdown(wait=True) method of ThreadPoolExecutor() class. When we look at the doc about shutdown(wait=True) method :

If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If wait is False then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing.

That's why your main thread blocked end of the with statement.

I want to give an answer to your second question, but i am confused with something about main thread exit or not. I will edit this answer later (for second question)

Python - Why ThreadPoolExecutor().submit a thread with queue blocks and Thread().start doesn't?

2 Answers