Prime new topic subscribers with old messages in Apache Artemis

Question

I'm configuring an Apache Artemis message-broker. The broker will accept big files and downstream consumers access the topic to process the latest files. Now I'm wondering how to make the latest files available for dev-runs. Because the messages only arrive a few times a day, the test runs would need to access the last few sent messages and can't wait for the next.

For production and staging-systems, I found that durable subscriptions work fine. I've adapted an Apache Camel config to serve as an illustration. Here are two consumers that receive messages, each using a durable subscription:

<route id="inbox">
    <from uri="file:inbox"/>
    <to uri="activemq:topic:testing"/>
</route>

<route id="outbox-staging">
    <from uri="activemq:topic:testing?clientId=staging&amp;durableSubscriptionName=staging"/>
    <to uri="file:outbox-staging"/>
</route>
<route id="outbox-production">
    <from uri="activemq:topic:testing?clientId=production&amp;durableSubscriptionName=production"/>
    <to uri="file:outbox-production"/>
</route>

This is fine. If a consumer is offline it will pick up messages when it comes back online. Now if another consumer joins for testing;

<route id="outbox-testing" streamCache="true">
    <from uri="activemq:topic:testing?clientId=my-local-consumer&amp;durableSubscriptionName=my-local-consumer"/>
    <to uri="file:outbox-local"/>
</route>

because the subscription didn't exist before, the consumer will have to wait for new messages. What I'm looking for is new subscribers to be immediately primed with the available messages. I found different names for the concept such as prefetchPolicy, consumerWindowSize, or "retroactive consumer". But it's unclear to me which terms apply to Apache Artemis and how to set them up because the examples mostly refer to Apache ActiveMQ.

How can a configure Artemis so that a consumer joining on a new subscription gets past messages?

Justin Bertram Justin Bertram · Accepted Answer · 2019-12-19T16:03:56

The prefetchPolicy doesn't apply to ActiveMQ Artemis. It's for ActiveMQ 5.x.

The consumerWindowSize does apply to ActiveMQ Artemis.

However, neither prefetchPolicy nor consumerWindowSize apply to this situation as they're both related to "flow control" and have nothing to do with putting "missed" messages onto a JMS topic subscription.

The "retroactive consumer" feature is for ActiveMQ 5.x. A similar feature (called "retroactive address") will be available in ActiveMQ Artemis 2.11. It was implemented as part of ARTEMIS-2504.

Therefore you have a few options:

Wait for ActiveMQ Artemis 2.11 to be released (should be released in January).
Build your own version of ActiveMQ Artemis based on the master branch which includes the retroactive address feature.
Modify your test environment so that new subscribers don't have to wait so long for messages (e.g. send them more frequently).

Prime new topic subscribers with old messages in Apache Artemis

1 Answers