ActiveMQ Topic Memory Usage

Question

I need to publish a large number of [header only] messages, each of which belongs to a unique category. Consider:

A.B.C.D.E.F

Ideally, all other considerations aside, I would like to publish each message on its own topic but this could result in thousands of topics. The benefit is a very simple wildcard based subscriber model. I am just not sure what the baseline memory usage is of a single topic and the aggregate usage of thousands of them.

For tuning, the topics will be non-persisted and non-durable. If possible, I could also limit the number of messages to 1 at a time, where an older message would be dropped. Messages will be published in waves every n seconds (usually > 15s). This would the reduce the memory usage.

My alternative is to publish to topics based on the first 2 segments like:

A.B

and put the rest of the segments in keyed headers. Subscribers would then have to use a combination of topic wildcards and selectors to subscribe to their desired feeds.

Anyone have any insight into this ?

Thanks !

//Nicholas

Answer 1

I setup a test to measure the actual memory consumption of topics. In a nutshell:

1. I start a loop with a base number of topics, created on the fly through a JMS session.
2. In an inner loop, I call System.gc() and then measure the heap space used. This is done 10 times with a 10 s. pause at the end of each inner loop.
3. I calculate the average heap usage sampled in the inner loop.
4. I increment the base number of topics by 100.

The results looked like this:

In summary, each topic consumed about 29.5 KB of heap space. Note that I did not expend much effort in tuning the topics' destination policies which might have reduced this somewhat.

Overall, I don't think this is bad, but I don't think it will scale to the numbers I want it to, so I am taking a different approach using Camel (as suggested by boday).

Basically, when a client subscribes using a topic wildcard, I am caching the pattern. When the Camel consumer receives a [non-jms] message, it checks the cache and if the message matches any of the wildcard entries in the cache, it will publish to the corresponding topic (which will be dynamically created). Once a topic stops receiving publications (because the cache had a matching wildcard removed), the topic will timeout and be "GC'ed".

Works fairly well.

//Nicholas