Take first message to arrive on one of 2 queues

Question

I have an application that I'd like to consume messages from queue A, but if queue A is empty, I'd like it to consume messages from queue B instead of sitting idle. I only ever want it to process 1 message at a time, and if the consumer is busy processing a message (from either queue), other pending messages (on either queue) should remain on their queue until the consumer is idle again (this gives other consumers a chance to process the new message).

So, for example, if a message arrives on either queue A or queue B while the consumer is idle (waiting), then the consumer picks it up. If a message arrives on either queue while the consumer is currently processing a message, nothing happens until the consumer has finished processing the current message. And if a message is available on both queue A and queue B, the consumer prefers to pick the message from queue A.

Edit: actually I'd be equally happy to be informed that messages are available to consume on both A & B, and implement my own logic to choose which one I actually want to pull from the queue.

Subscribing to both queues at once in separate threads doesn't appear to work because:

• if a message M arrives on queue A, the consumer will have to unsubscribe from queue B while it's processing message M, then resubscribe once message M is handled. Otherwise, if a message arrives on queue B, it might get picked up and dispatched simultaneously - I don't want to handle more than one message at a time.
• if a message arrives on both queues at once (or a message is already available on both queues when the consumer starts up or finishes processing the previous message), then it's possible that both messages will be picked up before the consumer has the chance to unsubscribe from queue B. If this happens I'd have to roll one of the messages back, but if this happens frequently the message will exceed the rollback count and get moved to a failure queue.

Ideally I'd like to generalise this to the N queues case. I'm interested in solutions that ideally work via JMS, but if I have to use the IBM MQ-specific API to do this that's OK too. Is there a usage pattern, or library, that will help achieve this? Are there alternative queueing approaches/techniques/technologies that allow it?

Pulling all messages off all queues and putting them onto a single queue is a no-go; we need to be able to clear individual queues independently, and different consumers may subscribe to different subsets of the N queues.

Answer 1

I don't know if this will be a big change for your code, but have you thought of adding Spring Framework and especially Spring Integration in your project? Such issues as you describe are already handled there.

This problem would be solved with two endpoints with event driven consumers on the two queues and a single thread pool.

http://www.eaipatterns.com/EventDrivenConsumer.html

http://en.wikipedia.org/wiki/Thread_pool_pattern

[Edit]

If you want the message to stay in the Queue until the thread is released you can follow the pattern of polling consumer

http://www.eaipatterns.com/PollingConsumer.html

which can chose as you wish from which queue to request a message first. Keep in mind that with this technique if your system processes messages faster than they arrive, you will experience some redundant polls to the queues.

Answer 2

If I were going to write this application, I think I would use an asynchronous consumer. I would set up two consumers using the MQCB verb both on the same connection handle (hConn) so only one would be driven at a time. Then start the consumers with MQCTL to set things going.

While you are handling the message within the call back function, you won't be driven again with a message from either queue.

To answer your additional question - if two messages arrive at the same time, there is no way to guarantee that Queue A's Callback is always preferentially driven. Clearly, if a message arrives on Queue B a split second earlier than Queue A, then your requirements are to process the message on Queue B. So that does raise the question why it matters if you process one message on Queue B before one message on Queue A when the arrive at exactly the same time.

I see from your edit in the question that you'd be happy just to be notified of messages having arrived and then you could go an do the gets yourself. You can also use Async Consume in browse mode, which means you could then be called whenever a message arrives on a queue, and then in the callback function you can issue the MQGET to retrieve the actual message with a destructive-get.

You could then have Queue B's callback function always do a quick no-wait MQGET call on Queue A before choosing to process it's own message to deal with the preferential requirement. This would be a bit inefficient, but definitely better than polling both queues because that adds wait intervals that you definitely don't want here. Whether the inefficiency of this is worth it will depend on your answer as to why it matters.

Queue A's callback function could just go straight to processing the message though.

Answer 3

My first reaction to this question would be to outsource the queue-handling to a Windows service (assuming you're running this on a Windows server). I don't think you would be able to do this entirely within MQ itself.

Set the triggering for each queue to First, and have the triggered application connect to the Service with information as to which queue it represents, and then terminate. The Service would then handle each queue according to whatever rules you set up for it.

Modifying answer according to questions in the comments:

1. By "within MQ itself" I mean that the describing functionality is probably not something you can accomplish by any setting on MQ components such as the queue manager, channels, queues and so on. You would have to set up some kind of servicing application outside MQ in order to do it.

2. "Why does it matter whether it is a service, a regular console application or a GUI?" It might not matter, but a Windows Service is always on, operates without user intervention, and can be interacted with by other attended and unattended applications. A console application or a GUI might be able to do some of this, but they're not designed for it, and may not work consistently. On the other hand, if you must interact with it, obviously a GUI or console app will be the way to go. I don't actually know what you're trying to do, however, so my ideas might be totally off-base for you.

3. "I'm not familiar with triggering - are you talking about this?" Yes. I've been working with Websphere MQ for years, and I've used triggering in virtually every application.

If you're working with a GUI, then probably you'll want to poll the various queues to see what they have and line them up in your GUI to be worked with, and none of my suggestion would have any relevance for you. I was assuming unattended operation, so there we are.