Messages "in delivery" status not released on ActiveMQ Artemis broker

Question

I have setup two ActiveMQ Artemis brokers (version 2.17) on a master-slave configuration with a shared file system for HA. During testing with heavy traffic I would stop master broker and see that the slave takes over and starts forwarding messages to consumers. After a while I can witness that a number of messages seem to have been stuck in queues. The "stuck" messages are considered as "in delivery" as I can see from the Artemis UI when queried.

My issue is that even when I restart the master broker these messages are not delivered to the consumer and remain stuck even if more messages are still populating the same queue and the queue has consumers. My assumption was that it had to do with previous connections setup by consumers still remaining active because there were not acknowledged.

So I did try to setup <connection-ttl-override> on broker, or on client connection string (tcp://host1:61616,tcp://host2:61616)?ha=true&connectionTtl=60000&reconnectAttempts=-1) but that did not seemed to have any effect since the connection would not be closed and the messages were not released. For consuming messages I am using Artemis JMS Spring client with a CachingConnectionFactory but also tried JmsPoolConnectionFactory to no avail.

<bean id="connectionFactory"
    class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory">
          <bean  class="org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory">
                 <property name="user" value="${spring.artemis.user}" />
                 <property name="password" value="${spring.artemis.password}" />
                 <property name="brokerURL" value="${spring.artemis.brokerUrl}" />
           </bean>
     </property>
     <property name="cacheConsumers" value="false"/>
</bean>

<int-jms:message-driven-channel-adapter
     connection-factory="connectionFactory"
     destination="myQueue"
     message-converter="messageConverter"
     channel="inputChannel"
     concurrent-consumers="${processing.poolsize}"
     max-concurrent-consumers="${max.processing.poolsize}"
     error-channel="errorChannel"
     acknowledge="transacted"
/>

The only remedy to this problem seems to be to restart consumer app which unblocks messages but that is not the desirable option.

How can I resolve this issue? Is there a way to release messages without manual intervention or restarting consumer app?

Is this something you can reproduce? If so could you provide a minimal reproducible example? — Justin Bertram
Sorry about that, I was conducting a number of tests to confirm if the solution suggested by Gary below solves my issue (using CachingConsumerFactory without variable concurrency and avoiding caching consumers). So far with the proposed configuration by Gary I was unable to reproduce that issue and no messages were blocked. — ypanag
No worries. If Gary's answer proves accurate with additional testing please don't forget to mark it as correct to help other users who hit this in the future. Thanks! — Justin Bertram

Gary Russell Gary Russell · Accepted Answer · 2021-03-03T15:05:57

Using variable concurrency with the CachingConnectionFactory could be the issue.

When an "idle" consumer is returned to the cache, the broker doesn't know the consumer is not active and still sends messages to it.

The caching factory is really only needed on the producer side (JmsTemplate) to avoid creating a connection and session for each send.

It's best to not use the CCF if you use variable concurrency; or configure it to not cache consumers, or don't use variable concurrency.

Messages "in delivery" status not released on ActiveMQ Artemis broker

1 Answers