When is the distributor storage queue get cleared

Question

I have a distributor and 2 workers, in distributor's storage queue messages are keep on increasing, each message has size of 2 kb.

I have gone through Udi reply here http://tech.dir.groups.yahoo.com/group/nservicebus/message/10698 Messages are cleared from the distributor's storage queue when it dispatches messages to the worker nodes. That being said, the worker nodes usually send a ready message right afterwards, causing the storage queue to fill up again.

I am not clear about “causing the storage queue to fill up again.” Statement.

I was expecting max messages in distributor's storage queue should be equal to number of workers but that is not the case.

Is there any configuration that I may be missing to control distributor’s storage queue clean up behavior?

Answer 1

As of NSB 3.3.2 there is an known issue, https://github.com/NServiceBus/NServiceBus/issues/806, with using SendLocal on worker in a distributor scenario. This will lead to the worker sending a "ready for a new message" every time it handles a message sent locally. This makes the distributors storage queue grow above the expected maximum of sum of all worker threads on all workers. This can be a problem if your sending lots of messages locally and the number of entries in the storage queue grows very large.

Workarounds:

1. Switch from SendLocal to Send
2. Restart the workers(this will reset the storage, but it will build up again for every SendLocal on a worker

There was also a similar issue with SLR(Second Level Retries) functionality( https://github.com/NServiceBus/NServiceBus/issues/571 ), but that was fixed in 3.2.7.

Answer 2

The queues will get cleared when the worker restarts. I'm guessing you are running under 2.x and I do recall that the number of messages in the queue is double that of the number of workers. I don't recall exactly why, but it does not affect anything really.

In 3.x I just validated that there is one message per worker, and they will be cleared when a worker is restarted. Once a restart occurs, the worker will adjust and report its availability right after, so the queue will have messages in it right after it comes back up.