activemq slow consumers block producer although producerFlowControl is false

Question

I have activemq used in my system and what i see is the following message: TopicSubscription: consumer=...: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@1684f89c] is full, temp usage (0%) or memory usage (100%) limit reached, blocking message add() pending the release of resources.

This is because if i understand correct my consumer is slow while my producer is fast. The result out of it is that eventually my producer is blocked untill consumer reads the message and frees some memory. What i whant is that my producer is not blocked and also when memory is full old messages are being discurded.

Given my understanding of what i have read the following configuration should do the trick (messageEvictionStrategy, pendingMessageLimitStrategy) but it is not working for me and i cannot figure out why.

I have specified low memoryusage limit low (35Mb) to make issue apear faster for testing reasons, but the case is that i need it eventually when the problem apears for activemq to just drop old messages.

I have found one non satisfactory solution of setting in ActiveMQConnectionFactory useAsyncSend=true and specifying sendTimeout. This makes producer not blocked but in this way the newest message is dropped and not the olderst one.

Finally, i am talking for non durable topics.

Any help guys would pe perfect. Below i have activemq configuration

        <destinationPolicy>
        <policyMap>
            <policyEntries>
                <policyEntry topic=">" producerFlowControl="false" memoryLimit="35 Mb">
               <pendingSubscriberPolicy>
                   <vmCursor />
                </pendingSubscriberPolicy>
                    <messageEvictionStrategy>
                        <oldestMessageEvictionStrategy/>
                    </messageEvictionStrategy>
                    <pendingMessageLimitStrategy>
                        <constantPendingMessageLimitStrategy limit="10"/>
                    </pendingMessageLimitStrategy>
                </policyEntry>
            </policyEntries>
        </policyMap>
    </destinationPolicy>

    <systemUsage>
        <systemUsage sendFailIfNoSpace="true">
            <memoryUsage>
                <memoryUsage limit="35 mb"/>
            </memoryUsage>
            <storeUsage>
                <storeUsage limit="1 gb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="5000 mb"/>
            </tempUsage>
        </systemUsage>
    </systemUsage>

activemq version 5.7.0

i use spring template to send messages:

    <bean class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
    <property name="timeToLive" value="100"/>
</bean>

I transmit javax.jms.ObjectMessage, relativelly small in size.

I found the problem in customer premisses I have many toppics in my application but managed to reproduce it loccally sending from 1 thread, non-stop messages continiusly always to the same topic. The message send was just a small string.

I have only one producer and problem seems to appear when i have 1 (or more) slow consumer(s) -but one slow consumer is enough-. if no slow consumer exists, problem does not appear.

I do not think it makes any difference but i use

       <transportConnectors>
       <transportConnector name="openwire" uri="nio://0.0.0.0:33029?wireFormat.maxInactivityDuration=60000&amp;wireFormat.maxInactivityDurationInitalDelay=60000"/>
    </transportConnectors>

difficult question or i am not describing something? please let me know if something is unknown or needs further clarification — Alexandros

ceposta ceposta · Accepted Answer · 2013-11-06T17:25:41

How can I recreate this? How many producers/consumers are attached to this topic? Is it only one topic?

Your settings look okay, but you don't need to set memoryLimit=35mb on the policy. It kinda doesn't make sense to set it the same as the overall system usage. The idea is that memory limit for all topics combined would equal the system memory limit. So for example, if you have two topics, each would use 35MB (2 * 35 == 70MB) and that would exceed the overall system memory settings. I don't think this is the specific cause of what you're seeing, but something to keep in mind.

Also what version of ActiveMQ is this? If you have some tests already written that can produce this, let me know.

activemq slow consumers block producer although producerFlowControl is false

2 Answers