ActiveMQ:'channel inactive for too long' exceptions stop broker messaging

Question

My system has the following parts:

ActiveMQ broker exposed on tcp, port 61616
3 Grails/Spring wars that live in their own Tomcat servers, they publish and consume messages to the JMS broker
n times remote client system with a JMS listener component to receive client specific messages, connect to the JMS broker through VPN using a hostname and port 61616

So far, all works fine throughout dev, Test and production environments.

We've just connected a new client system in production and we've noticed that it's logs start to report 'channel was inactive for too long' exceptions and drops the connection. Worrying the overall effect of this one client is that it stops all message consumption on the broker so brings are whole system to a halt.

This client listener (using Spring caching connection factory) appears to connect to the JMS broker ok, process some messages, then 3 mins reports the exception. Turned on DEBUG in ActiveMQ and got loads of output, nothing suggesting a warning or error on the broker around the same time though.

Believe that ActiveMQ has some internal keep alive that should keep the connection even if inactive for longer than the default 30 seconds.

Infrastructure guys have monitored the VPN of this client and confirm it stays up and connected the whole time.

Don't believe it is code or Spring config that is at fault, as we have numerous other instances of the listener in different clients and they all behave themselves fine.

Suppose I have 2 questions really:

What is causing 'channel inactive' exceptions?
Why does this exception in a single client stop ActiveMQ from working?

EDIT - adding exception stacktrace:

2013-04-24 14:02:06,359 WARN  - Encountered a JMSException - resetting the underlying JMS Connection (org.springframework.jms.connection.CachingConnectionFactory)
javax.jms.JMSException: Channel was inactive for too (>30000) long: jmsserver/xxx.xx.xx.xxx:61616
    at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
    at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1833)
    at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1850)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
    at org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:266)
    at org.apache.activemq.transport.InactivityMonitor$4.run(InactivityMonitor.java:186)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:693)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:719)
    at java.lang.Thread.run(Thread.java:813)
Caused by: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: jmsserver/xxx.xx.xx.xxx:61616
    ... 4 more

You'll have better luck with this on the ActiveMQ mailing list. The devs would need broker logs etc to investigate, no mention of broker version either. — Tim Bish
@shuttsy - wondering if you tried it and the results were the same. good luck! — ali haider
Found this: activemq.apache.org/activemq-inactivitymonitor.html — BPS
Did you able to fix this error? I am having similar one, sometimes even it is inactive more than 30 sec. connection is still alive. — Sahin Yanlık

ali haider ali haider · Accepted Answer · 2013-05-01T18:38:37

Have you tried the following:

Disable the InactivityMonitor; wireFormat.maxInactivityDuration=0 e.g.

URL: tcp://localhost:61616?wireFormat.maxInactivityDuration=0
If you don't wish to disable, have you tried setting it to a high number e.g.: URL: tcp://localhost:61616?wireFormat.maxInactivityDuration=5000000 (just an example - use your own time in ms)
Also, ensure that the jar files are the same version for both client and server.

Hope it helps

ActiveMQ:'channel inactive for too long' exceptions stop broker messaging

2 Answers