ActiveMQ Message Group Consumer Selection Across Multiple Queues?

Question

ActiveMQ Message Groups are a wonderful feature for load balancing across multiple consumers. In short: a stream of messages is partitioned across multiple consumers of a single queue according to a group identifier embedded in the message (JMSXGroupID). (So, consumer 1 will get all messages with JMSXGroupID = a, consumer 2 will get all messages with JMSXGroupID = b, and so on.)

Now, imagine you have 2 queues: A and B, and imagine that a consistent taxonomy of JMSXGroupIDs is used in messages flowing through both queues. Will the consumer the broker chooses for JMSXGroupID = ABC on queue A be the consumer from the same connection that the broker chooses for JMSXGroupID = ABC on queue B?

I suspect the answer to the question as I've asked it is "no." There are too many variables in play: What happens if the consumer the broker chooses for A has no corresponding consumer for B? What happens if the consumer the broker chooses for A has multiple corresponding consumers for B? There's no obvious right answer in these cases.

However, can we simulate this behavior? For example, a consumer on a composite destination could be a viable solution -- make sure all consumers on A and B consume on the composite destination A,B and you might be in business -- but ActiveMQ does not appear to support consuming from composite destinations.

The only solution I've found is simply to push messages for both A and B on one single queue -- call it AB -- and have an exclusive consumer on that. You now have to distinguish between "A messages" and "B messages," but you could easily do that with headers.

However, this solution smells funny. (You now have to assume producers will dutifully apply special headers to their messages, or modify your payloads.) Is there a solution that will make sure consumers across two separate queues A and B always land on the same connection?

Jakub Korab Jakub Korab · Accepted Answer · 2014-01-09T11:25:26

As you have correctly worked out, message groups apply to a single queue only. There is no coordination between multiple queues.

Generally when you are using message groups you are trying to guarantee message ordering across not just delivery, but processing - so that say all events for a particular entity are processed in sequence. The devil is always in the details of your use case, but putting all of the related messages onto a single queue will give you the result that you are after. In order to process them differently, you then need to put some sort of multiplexing logic into your consumer to make the decision based on the message payload - a well known header as you say is a good candidate for a solution.

To get around the prerequisite of ensuring that clients explicitly set this, what you can do is write a piece of Camel routing logic that does this on your behalf - this is only possible with the broker: component that was added to ActiveMQ 5.9. The idea will be that producers see two separate queues - A & B; the routing logic will read from those queues as the messages are being put in, set the header appropriately, and re-route them to C instead. The routing logic in effect works as an interceptor.

<route id="ConflateA">
  <from uri="broker:queue:A"/>
  <setHeader headerName="OriginalMessageSource">
    <constant>A</constant>
  </setHeader>
  <to uri="broker:queue:C"/>
</route>

You can then use the OriginalMessageSource header in your multiplexing logic.

ActiveMQ Message Group Consumer Selection Across Multiple Queues?

1 Answers