ActiveMQ Topic to Queue, When using Camel route messages getting lost when switching from Master to Slave?

Question

The setup: Default ActiveMQ.XML. One ActiveMQ instance per server on 3 Servers at Local data-center, One ActiveMQ instance at a remote data-center Server. All 3 + 1 instances are running, but only one ActiveMQ instance is the master at any given moment per data-center. Messages from all instances at all data-centers are persistent to a network KahaDB, and we've configured two retries per message.

The goal: To keep the queues synchronized between data-centers.

The problem: To test when the remote server is down, that all messages will be saved/persisted until the remote server is back online. I will send three messages to an ActiveMQ Topic.

A camel route is configured to read from the topic and push to 2 local queues.

A second camel route is setup to read from one of the local queues, and push the message to a remote ActiveMQ.

When the remote server is off (data-center 2 is down), and we fail over from the Master ActiveMQ (by stopping the service) to a slave, 1 out of the 3 messages is lost to the ether. It seems to be the first message that's sent to the remote server. As if it's getting a connection refused or something, and it disappears forever, not even going into the DLQ?

The camel configuration:

<bean id="local" class="org.apache.activemq.camel.component.ActiveMQComponent">
  <property name="brokerURL" value="tcp://localhost:12345"/>
  <property name="preserveMessageQos" value="true" />
</bean>

<bean id="remote" class="org.apache.activemq.camel.component.ActiveMQComponent">
  <property name="brokerURL" value="failover:(tcp://remotehost:54321)"/>
  <property name="preserveMessageQos" value="true" />
</bean>

<camelContext id="topicToQueueCamelContext" xmlns="http://camel.apache.org/schema/spring">
    <route id="topicToQueue">
        <from uri="local:topic:SomeTopic"/>
        <to uri="local:queue:SomeQueue"/>
        <to uri="local:queue:SomeQueue.Remote"/>
    </route>
</camelContext>

<camelContext id="queueToRemoteQueueCamelContext" xmlns="http://camel.apache.org/schema/spring">
    <route id="queueToRemoteQueue">
        <from uri="local:queue:SomeQueue.Remote"/>
        <to uri="remote:queue:SomeQueue"/>
    </route>
</camelContext>

Tried: Setting the remote camel routes as transacted, and setting trackMessages=true on the brokerURL Failover Transport.

Does anyone have any ideas on which ActiveMQ BrokerURL, ActiveMQ XML, or Camel URI Query String Parameters might be missing -- or any ideas on how to solve or debug this missing message problem?

burki burki · Accepted Answer · 2019-01-04T08:30:09

You need to configure your ActiveMQComponent to consume messages transactional. By default message consumption is not transactional. That way you get the best performance, but you also "accept" to lose messages.

ActiveMQ Topic to Queue, When using Camel route messages getting lost when switching from Master to Slave?

1 Answers