Permanent JMS Connection failure but no error or InactivityExceptions

Question

We have a problem similar to this old question. Our setup, however, is a little different. For instance, heartbeats should already be present as we have the default InactivityMonitor from ActiveMQ.

We have a client which uses an embedded broker. The embedded broker has a network connector to a remote broker running as a standalone service on the machine. In this way, we can decouple the communication between client and server. The embedded broker serves as a local queue for the client.

The client sends messages to the embedded broker. These message either flow via the network connector to the remote broker or (when the connection is temporarily unavailable) stay in the embedded broker until the connection is reestablished.

Both the embedded broker and the remote broker are instances of Apache ActiveMQ. The JMS implementation is based on Spring JMS.

In practice we sometimes see strange behavior (usually after a long period without any issues):

the network connector is listed in the connections-tab of the management console of the remote broker. However, not all messages are being delivered at the remote broker. Usually bytesmessages are stuck in the embedded broker, while text messages are delivered to the queues on the remote broker.
the network connector is listed in the connections-tab of the management console of the remote broker. However, no messages are being delivered at the remote broker.

The inactivity-monitor is enabled on the remote broker. The embedded broker is created using the code shown below (SSL code is omitted for brevity).

We have no idea what might cause the problem and, more importantly, why the setup does not recover automatically. We expected this setup to detect automatically that a connection is not available and setup a new connection. This does not seem to be happening however.

Does anyone have an idea where we could start looking or, even better, have an idea what the problem might be?

@Bean
public ActiveMQConnectionFactory activeMQConnectionFactory(final Optional<TransportListener> transportListener)
{
    final ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory();

    activeMQConnectionFactory.setClientIDPrefix(clientIDPrefix);
  activeMQConnectionFactory.setBrokerURL(this.env.getProperty("activemq.connection.url"));
    activeMQConnectionFactory.setUserName(this.env.getProperty("activemq.users.username"));
    activeMQConnectionFactory.setPassword(this.env.getProperty("activemq.users.password"));

    transportListener.ifPresent(activeMQConnectionFactory::setTransportListener);

    return activeMQConnectionFactory;
}

@Bean
public CachingConnectionFactory cachingConnectionFactory(final ActiveMQConnectionFactory activeMQConnectionFactory)
{
    return new CachingConnectionFactory(activeMQConnectionFactory);
}

@Bean
public JmsTemplate jmsTemplate(final CachingConnectionFactory cachingConnectionFactory)
{
    final JmsTemplate jmsTemplate = new JmsTemplate(cachingConnectionFactory);
    return jmsTemplate;
}

@Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(final ActiveMQConnectionFactory activeMQConnectionFactory)
{
    final DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
    factory.setConnectionFactory(activeMQConnectionFactory);
    factory.setConcurrency("3-10");

    return factory;
}

@Bean
public MessageSender messageSender()
{
    return new MessageSender();
}

@Bean(initMethod = "start", destroyMethod = "stop")
public BrokerService bufferingBroker() throws Exception
{
    final BrokerService broker = new BrokerService();

    String brokerName = UUID.randomUUID().toString();
    broker.setBrokerName(brokerName);

    final NetworkConnector networkConnector = new DiscoveryNetworkConnector(
        new URI("static://" + this.env.getProperty("activemq.connection.url")));
    networkConnector.setUserName(this.env.getProperty("activemq.users.username"));
    networkConnector.setPassword(this.env.getProperty("activemq.users.password"));
    networkConnector.setNetworkTTL(5);
    broker.addNetworkConnector(networkConnector);

    return broker;
}

@Bean
public ActiveMQConnectionFactory embeddedActiveMQConnectionFactory(final BrokerService bufferingBroker) throws Exception
{
    final ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory();
    activeMQConnectionFactory.setBrokerURL("vm://" + bufferingBroker.getBrokerName() + "?create=false");

    return activeMQConnectionFactory;
}

@Bean
public CachingConnectionFactory embeddedCachingConnectionFactory(final ActiveMQConnectionFactory embeddedActiveMQConnectionFactory) throws Exception
{
    return new CachingConnectionFactory(embeddedActiveMQConnectionFactory);
}

@Bean
public JmsTemplate embeddedJmsTemplate(final CachingConnectionFactory embeddedCachingConnectionFactory) throws Exception
{
    return new JmsTemplate(embeddedCachingConnectionFactory);
}

@Bean
public DefaultJmsListenerContainerFactory embeddedJmsListenerContainerFactory(final ActiveMQConnectionFactory embeddedActiveMQConnectionFactory) throws Exception
{
    final DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
    factory.setConnectionFactory(embeddedActiveMQConnectionFactory);
    factory.setConcurrency("3-10");

    return factory;
}

@Bean
public BufferedMessageSender bufferedMessageSender()
{
    return new BufferedMessageSender();
}

Pieter Pieter · Accepted Answer · 2019-11-27T15:23:41

It turned out an external firewall was blocking bytesmessages whereas textmessages could pass through without any issue.

Permanent JMS Connection failure but no error or InactivityExceptions

1 Answers