Mule Persistent ActiveMQ RedeliveryPolicy

Question

I am using Mule as ESB solution. I have a queue, from where i am getting messages and trying to make http request to service that is failing all the time.

I have configured the RedeliveryPolicy on ActiveMQ like this:

    <spring:bean id="retryRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy"
        name="retryRedeliveryPolicy">
        <spring:property name="maximumRedeliveries" value="76" />
        <spring:property name="initialRedeliveryDelay" value="300000" />
        <spring:property name="maximumRedeliveryDelay" value="3600000" />
        <spring:property name="useExponentialBackOff" value="true" />
        <spring:property name="backOffMultiplier" value="2" />
        <spring:property name="queue" value="*" />
    </spring:bean>

It retries after 5min. then 10min ,20,40,60,60,60... For about ~ 3days

The problem is, that Retry logic is non persistent.

Lets say, message was retrying for 2 days. And i have deployed new version of mule app, or restarted server... In this case, retry logic will start all over again from 5min, 10min.... Because retry state is kept in RAM memory by the Client.

Hot to make RedeliveryPolicy persistent? It must keep retrying for 1 more day after i will restart the server after 2 days.

One solution i think might be to set timeToLive for 72 hours for message. But even then after restarting server. It will not retry every hour from start. It will start from 5min...

Answer 1

You can use the JMSXDeliveryCount property of the JMS message to check how many times it has been retried already. The retry time interval logic should rely on the JMSXDeliveryCount variable.

message.getIntProperty("JMSXDeliveryCount ")

http://activemq.apache.org/activemq-message-properties.html

Answer 2

ActiveMQ has a way to do persistent redelivery, but it's not built in using the RedeliveryPolicy on the ConnectionFactory, which is intended for short periods of redelivery before failing.

Persistent redelivery can be built using the the ActiveMQ scheduler and delayed messages however. A bit manual, but doable. Make sure you turn on schedulerSupport="true" in ActiveMQ before trying this.

A JMS property: "delivery" keeps track of number of retries and if things go wrong a catch exception strategy redelivers the message a number of times with a delay. The actual delay is handled by ActiveMQ, so this flow can handle delays of hours, days etc and withstand restarts of both ActiveMQ and Mule.

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:scripting="http://www.mulesoft.org/schema/mule/scripting"
    xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata"
    xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/current/mule-scripting.xsd">

    <spring:beans>
        <spring:bean name="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
            <spring:property name="maximumRedeliveries" value="0" />
        </spring:bean>

        <spring:bean name="cf"
            class="org.apache.activemq.ActiveMQConnectionFactory">
            <spring:property name="brokerURL" value="tcp://localhost:61616" />
            <spring:property name="redeliveryPolicy" ref="redeliveryPolicy" />
        </spring:bean>
    </spring:beans>

    <jms:activemq-connector name="Active_MQ"
        specification="1.1" brokerURL="tcp://localhost:61616"
        validateConnections="true" doc:name="Active MQ" numberOfConsumers="1"
        maxRedelivery="-1" connectionFactory-ref="cf" />

    <flow name="testFlow">
        <jms:inbound-endpoint queue="IN" connector-ref="Active_MQ"
            doc:name="JMS">
            <jms:transaction action="ALWAYS_BEGIN" />
        </jms:inbound-endpoint>

        <set-property propertyName="delivery"
            value="#[message.inboundProperties['delivery'] or 0]" doc:name="Property" />

        <scripting:component doc:name="Throw Error">
            <scripting:script engine="Groovy"><![CDATA[throw new java.lang.RuntimeException("Error in processing")]]></scripting:script>
        </scripting:component>

        <choice-exception-strategy doc:name="Choice Exception Strategy">
            <catch-exception-strategy doc:name="Catch Exception Strategy"
                when="#[message.outboundProperties['delivery'] &lt; 5]">
                <logger level="ERROR"
                    message="Retry once more count #[message.outboundProperties['delivery']]"
                    doc:name="Logger" />

                <set-property propertyName="AMQ_SCHEDULED_DELAY" value="3000"
                    doc:name="Property" />
                <set-property propertyName="delivery"
                    value="#[message.outboundProperties['delivery'] + 1]" doc:name="Property" />
                <jms:outbound-endpoint queue="IN"
                    connector-ref="Active_MQ" doc:name="JMS">
                    <jms:transaction action="ALWAYS_JOIN" />
                </jms:outbound-endpoint>

            </catch-exception-strategy>

            <rollback-exception-strategy doc:name="Rollback Exception Strategy">
                <logger level="ERROR" message="Giving up retry. Do whatever needed here." doc:name="Logger" />
            </rollback-exception-strategy>
        </choice-exception-strategy>
    </flow>
</mule>

Answer 3

This is no way to make the RedeliveryPolicy persistent - it's controlled by the connection factory and the factory will get reset when you restart the server. The way you have it configured, it will carry on with the behaviour you see, as you have useExponentialBackOff set to true. You could set this to false to get the delay to be a regular amount, but that is the only change to make.

I think you have the right idea with setting the message TTL, at least this will remove the message after the specified time. The JMSXDeliveryCount is great, but it will remove after a number attempts, not a defined time period, if you increase the delay between the retries.