JMS topology (Queue with multiple consumers) and message groups

Question

here is a simplified / schematic topology we set up

   [Front server 1 (message producer)]  [Front server n (message producer)]
                   |                                    |        
                   |________________    ________________| 
                                    |  |         
                         [Messaging Server (HornetQ)]
                                    |  |
                    ________________|  |________________
                    |                                   |
   [Task server 1 (Message Driven Bean)]          [Task server n (MDB)]

Each node (server) is a standalone (no cluster) jboss application server (Jboss-as7) including the messaging one.
The messaging server deploys many JMS queues.
Each Task server deploys a MDB per queue with many consumers.
All message producers use the same inbound adapter, all message consumers (MDBs) the same outbound one. In fact all front node are exactly the same (same AS, same config, same deployed artifacts), same for all server nodes.

Now here is my problem :

the application is a multi-tenant one, for a given queue, some tasks (message processing) must not be processed in parallel for a given tenant, we set up so message grouping to handle this constraint. The message group is the tenant name and is set by the message producer when sending the message :

message.setStringProperty("JMSXGroupID", tenantName);

On a platform we have 1000+ tenant (so 1000+ different message groups) and for a given queue 3 consumers per server and 3 tasks servers.

When monitoring this queue on the messaging server we can so see thousands messages in queue and 9 consumers. We expects message to be delivered 9 by 9 but in practice the in-delivery message count never exceed 1.

What's the problem here ? is the topology suitable for our needs ?

Gab Gab · Accepted Answer · 2015-11-09T11:14:32

Answer from jboss hornetq support forum (crédits Justin Bertram):

As I'm sure you're aware, a queue has first-in-first-out (i.e. FIFO) semantics. Grouped messages basically act like a bottleneck on the queue since they ensure serial message processing. Here's a simple example...

Consider groups A, B, and C which contain two messages each for a total of 6 messages in the queue. Also consider that they are in the order A, A, B, B, C, C. Now consider 3 different consumers each consuming a different group. The consumer for group A will receive the first message, process it, and acknowledge it so that it is removed from the queue. Then it will receive the second message, process it, and acknowledge it so that it is removed from the queue. During the time the consumer for group A is busy with these 2 messages no other messages can be consumed from the queue. Only when the second message is acknowledged can the consumer for group B actually receive the first message in group B. Once the consumer for group B acknowledges both of its messages the consumer for group C can finally receive the messages in its group. This behavior is governed by the FIFO semantics of the queue. The messages in group B can't leap-frog the messages in group A and be consumed before all the messages in group A are consumed. The same goes for the messages in group C.

Since all your messages are in a group I would never expect the in-delivery count to exceed 1.

JMS topology (Queue with multiple consumers) and message groups

1 Answers