activemq-cpp throws "Thread local storage limit" reached when sending

1
votes

I have been developing a threaded data pooling system interconnected with ActiveMQ-cpp (library version 3.8.2). The system manages a group of devices which can be pooled periodically, the poll results sent to a queue in the broker. The answers queue can be changed in any time by a control message, so the producers and destinations are constantly changed. Only one subscriber receives all the control commands, and several threads create producers and messages from their own sessions. All messages are TextMessages.

All activemq-cpp elements are wrapped around in RAII elements, yet at some point after intensive usage, the connection receives messages but when I try to send a response the library throws an exception reading "Thread Local storage limit reached". Afterwards it receives a few more messages (sometimes text is broken) but trying to emit new messages fall again in the Thread storage exception and then stops receiving and sending entirely.

Has anybody found this kind of issue? Any help is welcome.

Update: I have compiled and used the new 3.8.3 version and can still reproduce the error. Tests are done in RHEL 5.7 and Fedora 20, both 64 bits architectures.

1
c++multithreadingactivemq-cpp
are you using a lot of variables on the thread stack? – Matt
@Matt 99% if the variables in the system are shared RAII objects in the process stack, the 1% remaining are the scoped producers/messages. – Nare
Try moving to version 3.8.3 to see if a fix there solves it – Tim Bish
@TimBish: Still breaks using 3.8.3, updated the question to reflect that. – Nare
@Nare TLS's behaviour is OS (and thread library) dependent, which one are you using? – James

1 Answers

4
votes

After several days of debugging and refcounting in the wrappers we found out the culprit was a misplaced pointer to a structure that wasn't deleted. This in particular hid the missing pointer from Valgrind's memcheck as an "indirect loss".

The issue at hand was, after all, a collection of undeleted cms::Session pointers (would also happen with other cms objects).

As the activemq-cpp library points out in the Session section the cms::Session is a single thread element. However, it also states that the Connection which created it controls several actions of it, for which it keeps a reference to the generated Sessions among other things, all this stored in the Connection thread stack. Thus a massive amount of Session creations will trigger the "Thread Local storage limit reached" when the Connection capacity is full.