Python3 queue: When can a blocking wait with no timeout return None?

Question

I am wondering when a blocking get with no timeout on a python3 queue can return None.

Queue.get(block=True, timeout=None)

Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

For me this means that a get() with no arguments will wait on an element being in the queue and return only then, therefore always will return a value that is not None. Still in the example at the bottom of the queue documentation the following code is given:

while True:
    item = q.get()
    if item is None:
        break
    ...

The explicit checking against the item being None implies that None can be returned. In which situations can this happen?

It's because they put() a None into the queue explicitly as an end marker (which, by the way, is a good technique to signal to the partner thread that no more communication will happen on the queue). — blubberdiblub
Oh, thanks. Should have looked at the example more carefully... — MartinM

blubberdiblub blubberdiblub · Accepted Answer · 2017-05-03T11:23:53

Frequently, ending threads properly in a clean way is a non-trivial problem in multi-threaded programs.

When the main communication channel between your threads is a queue, which the consuming thread will usually block on while calling get(), a good technique is using a value that cannot normally occur as data to signal that no more data is going to be enqueued and that the consumer thread is supposed to exit.

Producer thread:

with open('/run/myapp/datapipe', 'r') as f:
    for line in f:
        queue.put(line)

# signal end of communication with a sentinel value of None
queue.put(None)

# further clean up

Consumer thread:

while True:
    line = queue.get()
    if line is None:
        # if sentinel value was found, break and clean up
        queue.task_done()
        break

    # process line here
    queue.task_done()

# clean up

In case None can occur as part of the normal data stream, you could also come up with another suitable value as end of communication marker.

Also, if you have multiple consumer threads, this simple end marker enqueuing technique won't suffice, as only one of the threads will dequeue the marker. In that case, you could make each consumer thread put() the marker back into the queue once it fetched it, thereby handing the end marker through each consumer thread one-by-one.

Python3 queue: When can a blocking wait with no timeout return None?

1 Answers