Azure Service Bus : Amqp Idle Timeout condition = amqp:link:detach-forced

Question

The error I get:

2019-12-09 06:39:33.189 ERROR 107132 --- [http-nio-8082-exec-5] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.jms.IllegalStateException: The MessageProducer was closed due to an unrecoverable error.; nested exception is javax.jms.IllegalStateException: The MessageProducer was closed due to an unrecoverable error.] with root cause

javax.jms.JMSException: Idle link tracker, link qpid-jms:sender:ID:7300953e-f587-4ae3-b9fe-85b84e032554:1:101:1:order-update has been idle for 1800000ms TrackingId:801ab247-3f36-4470-8665-08846eb1c181_G24, SystemTracker:client-link34404815, Timestamp:2019-12-06T21:04:35 [condition = amqp:link:detach-forced]
    at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToException(AmqpSupport.java:164)
    at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToException(AmqpSupport.java:117)
    at org.apache.qpid.jms.provider.amqp.AmqpAbstractResource.processRemoteClose(AmqpAbstractResource.java:262)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:906)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:102)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider$17.run(AmqpProvider.java:792)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

dependencies

compile group: 'com.microsoft.azure', name: 'azure-servicebus-spring-boot-starter', version: '0.2.0'
compile group: 'javax.jms', name: 'javax.jms-api', version: '2.0.1'
compile group: 'org.apache.qpid', name: 'qpid-jms-client', version: '0.28.0'
compile group: 'org.apache.camel', name: 'camel-jms', version: '2.24.1'
compile group: 'org.springframework.integration', name: 'spring-integration-jms', version: '5.0.4.RELEASE'

jmsConnectionFactory configuration:

<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory">
        <bean class="org.apache.qpid.jms.JmsConnectionFactory">
            <constructor-arg value="${azure.jms.url}" />
            <property name="username" value="${azure.jms.username}" />
            <property name="password" value="${azure.jms.password}" />
            <property name="clientID" value="AltaPay" />
            <property name="receiveLocalOnly" value="true" />
            <property name="localMessageExpiry" value="true" />
            <property name="populateJMSXUserID" value="true" />
        </bean>
    </property>
    <property name="exceptionListener">
        <bean class="com.lauraashley.microservice.altapay.callback.exception.CustomJMSExceptionListener" />
    </property>
    <property name="sessionCacheSize" value="10" />
    <property name="cacheConsumers" value="false" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
    <property name="connectionFactory" ref="jmsConnectionFactory" />
    <property name="cacheLevelName" value="CACHE_NONE" />
</bean>

CustomJMSExceptionListener

public class CustomJMSExceptionListener implements ExceptionListener {

    private static final Logger logger = getLogger(CustomJMSExceptionListener.class);

    @Override
    public void onException(JMSException exception) {
        // TODO Auto-generated method stub
        logger.error("--------------- Catched exception with CustomJMSExceptionListener ---------------");
        logger.error("Error code:"+exception.getErrorCode());
        logger.error("Msg:"+exception.getMessage());
        exception.printStackTrace();
        logger.error("---------------------------------------------------------------------------------");
    }
}

How I reproduce it

First: the CustomJMSExceptionListener is not used, isn't configured ok?

The application is a eccomerce app on OCC ( oracle cloud commerce ) platform that used java spring-boot services for the payment integration and flow. This error occurs when an order exceds the idle time then the connection with the Azure Service Bus fails and in order to reconnect I must restart the java app, and this is quite a big problem because no more orders can be processed. I read that CachingConnectionFactory has reconnectOnException wich by default is true.

I don't really understand why this happens and what is the solution in order to fix it.

Tim Bish Tim Bish · Accepted Answer · 2019-12-09T20:55:14

The exception is indicating that Azure has closed the producer because it was idle for to long, meaning it hadn't sent a message within the timeout (some documentation here). You might be able to work around this when using the CachingConnectionFactory by configuring the cache producers option to false so that producers are created on demand but I'm not entirely sure on that as I don't have any way to test it.

This isn't a Qpid JMS client level bug but rather the behaviour of Azure kicking in where after I think it's ten minutes of no activity on a link it will forcibly close the link. In a non-spring based application you'd have to account for this by catching the JMSException on send either attempting to create a new producer and sending again or by tearing down the whole connection and starting over. Your reaction somewhat depends on foreknowledge that you are using Azure and knowing that this can happen.