Tracking where rogue AMQ messages coming from

Question

AMQ 5.7 here. I have inherited a very old set of Java applications that use an ActiveMQ (AMQ) broker to communicate between each other as well as with clients outside this ecosystem. Hence the AMQ broker has several hundred (!!!) queues on it, and there are literally hundreds of JMS clients living on dozens of different servers (each) publishing messages to and consuming messages from these queues. Its a rats nest.

I have a queue, let's call it shouldBeDead, that should not be receiving any more messages. It was deprecated at some point in time and nothing should be sending messages to that queue anymore. However periodically it gets bombarded with hundreds of messages from something/somewhere. It has no consumers on it (which is correct; my suite of Java apps no longer use it anywhere in their code and so nothing is listening on that queue to consume messages off of it).

To complicate matters even more, this is an old AMQ version suffering from this known UI bug where the TLDR; is: I need to upgrade the AMQ instance to > 5.12.x. However for reasons outside the context of this question, I can't upgrade AMQ at this time. So whereas I was hoping to browse the enqueued messages on shouldBeDead and drill down into them to get information about them, I can't even view them in the AMQ web UI nor in the app logs.

I'm simply trying to figure out where these messages are coming from!

Doing a network analysis might be fruitful but is outside the scope of my skill sets and these messages flood the queue at seemingly random times without any discernable pattern.

I'm hoping there's some AMQ command-line fu I can do to inspect queue metadata, perhaps peek into KahaDB or any other type of magic I can do to look at these messages and/or get tracking/client-side info out of them.

Worst case I know I can deploy some code changes to add a shouldBeDead listener/consumer back in and to log the messages, however I'm really trying to do this without making any code changes. Any ideas/thoughts? Thanks in advance!

Answer 1

I would approach the problem via the JMX console.

Here is a list of diagnostic informations available via JMX, and here is a way how to access this data via command line. By knowing the exact details of your problem, like:

• how predictable these bursts are
• roughly how many messages are being
• sent how many connections you have

you need to capture the relevant info to catch your producer.

Without knowing the details of course, one idea that might work:
Run a bash script periodically that checks the size of the queue (like Destination/EnqueueCount) + periodically saves all the Active connections. When you detect the spike in the messages on the deprecated queue, look back what connection just appeared at that time (Connection/RemoteAddress).