Clients stuck at creating session

Question

I have a network of 3 brokers running but sometimes the brokers fail in a rather unique and annoying way.

They still accept connections but otherwise stop to communicate with the client (even those still connected). For new connections created in java this means:

con = factory.getConnection(); // method returns, connection is created
con.createSession(false, Session.AUTO_ACKNOWLEDGE); // never returns

on the server side there are no logged exceptions when running in debug mode.

do you have any idea what happens here? is there any logmessage i can look for?

EDIT: some additional info:

http://pastebin.com/9iztG67D - xml config file
every node is a master node with a connected slave (pure master slave)
client uri: failover:(tcp://serverA:61616,tcp://serverB:61616,tcp://serverC:61616,tcp://serverA-Slave:61616,tcp://serverB-Slave:61616,tcp://serverC-Slave:61616)?randomize=false

this is most probably not the case but just to make sure, do you free space/ram/cpu... etc on the BOX? — Eugene
Are you using failover in the client connection URI? Please add some more information on the client configuration. — Tim Bish
@Laures Hi, I am in great need of your help. Please see this question that is for integration of jersey rest service and Activemq. stackoverflow.com/questions/19706788/… — Kumar

jkysam jkysam · Accepted Answer · 2012-02-02T14:07:22

Without knowing more about how your brokers are configured and what kind of queue/topic structure you are using it's a bit difficult to tell exactly what is happening. But based on the general behavior that you described, this sounds like flow control at some level.

Look here for a more detailed explanation of flow control. Basically, if you have a destination that is filling up and hitting its memory limit due to a slow/hanging consumer the broker might give the appearance of "locking up" when new producers attempt to send messages.

Brokers might also lock up in strange ways when system memory limits are reached. Check out your configuration values for:

<systemUsage>
  <systemUsage>
    <memoryUsage>
      <memoryUsage limit="64 mb" />
    </memoryUsage>
    <storeUsage>
      <storeUsage limit="100 gb" />
    </storeUsage>
    <tempUsage>
      <tempUsage limit="10 gb" />
    </tempUsage>
  </systemUsage>
</systemUsage>

Easiest way of identifying this kind of problem is to use a JMX client, such as jconsole, connect to the brokers, and inspect the current usage of memory in relation to their limits, for different destinations and the overall broker.

Clients stuck at creating session

2 Answers