python multiprocessing queue get([block[, timeout]]),block the item in queue Or block the queue?

1
votes

get([block[, timeout]])
Remove and return an item from the queue. If optional args block is True (the default) and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Queue.Empty exception if no item was available within that time. Otherwise (block is False), return an item if one is immediately available, else raise the Queue.Empty exception (timeout is ignored in that case).

As document above.I write a program,just one producer process and six consumers.A queue share between the processes.

  • The producer use the method:put_nowait()
    1process * 6000 items/second
  • While the consumer use the get_nowait(),the consumer's get_nowait is very slowly.
    6 process * (0 ~ 500)items/second.
  • while the consumer use the get(True,1),the program works well.
    6 process * (1000 ~ 1600)items/second.

In my opinion. the get_nowait() should more faster the get(block) How does the block of the get() method work, can anyone give me some advice.

1
pythonqueuemultiprocessing

1 Answers

2
votes

There is no difference in performance between the get and the get_nowait. What changes is just its behaviour. This is well described in the documentation.

The get_nowait is equivalent as calling:

Queue.get(False)

This means the caller won't get blocked waiting for new data being available. It won't make the caller any "faster".

You use the get_nowait method if you want to opportunistically consume something if available and do something else otherwise.

The actual speed difference you experience might be due to the fact that non blocking get makes lots more accesses to its internal Lock compared to the blocking one. This will adds its cost.

You can see the actual implementation of the get_nowait here.