ActiveMQ 5.10/ QPid 0.28/ AMQP 1.0 performance issue

Question

I am trying to do a performance benchmark of using Openwire and AMQP with ActiveMQ and getting huge variation in throughput

Using Openwire

Persistent Message size: 43 byes, no compression, 200 concurrent connections, throughput around 9006 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, throughput around 3678.86 msgs/sec.

There is not much load on CPU less than 5% so I can use compression to get better throughput but that's a different story.

Using AMQP 1.0

Persistent Message size: 43 byes, no compression, 200 concurrent connections, throughput around 12.8 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, throughput around 11.9 msgs/sec.

Our configuration is as

**Client**
    - JMeter 2.11 using Apache QPid 0.28 as AMQP 1.0 client
    - Java 1.7
    - Ubuntu 12.04 LTS Quad Core 2.0 GHz,7GB RAM, 512 KB Cache

**Broker**
    - ActiveMQ 5.10 with KahaDB, using LevelDB didn't give much different numbers
    - Java 1.7
    - Ubuntu 12.04 LTS Quad Core 2.0 GHz,7GB RAM, 512 KB Cache

Tuning Efforts:

I looked at the following

For Openwire, on Broker side, changed tcp to nio

<transportConnector name="openwire" uri="nio://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.tcpNoDelayEnabled=true&amp;wireFormat.maxFrameSize=104857600"/>

On Client side url was as
    tcp://<broker ip>:61616?jms.useAsyncSend=true

For AMQP 1.0, on Broker side, changed to nio and added some options on url but have no apparent effect

<transportConnector name="amqp" uri="amqp+nio://0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>

On Client side
    connectionFactory = amqp://username:password@<broker ip>:5672?remote-host=default

All the tuning options available on Openwire are not available on amqp, especially on producer using jms.useAsyncSend=true gave me huge performance boost, of course with the less reliability of ack. I came to know that by default sending message with amqp is in async mode by default. Apparently the numbers are telling us that it is possibly being processed in synchronous mode ?

Here are some of the links that I already look into

• http://activemq.apache.org/amqp.html
• http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/
• http://activemq.2283324.n4.nabble.com/Interesting-Persistent-Messaging-Performance-td4676001.html
• http://activemq.2283324.n4.nabble.com/Any-Performance-Latency-benchmarks-for-ActiveMQ-s-AMQP-implementation-against-OpenWire-td4674713.html
• http://www.slideshare.net/ceposta/activemq-performance-tuning#

• http://working-with-activemq.blogspot.com/2012/05/performance-improvements.html

  Are there configuration (for connections or channels) or tuning options with ActiveMQ AMQP and/or QPid that I are unaware of ?

UPDATE 1:

As Tim pointed out below, my comparisons were incorrect, I was using Async.Send for Openwire and Sync.Send for AMQP 1.0, it was my mis-understanding that AMQP 1.0 by default uses Async.Send. Robbie pointed out that is not the case for Persistent messages. Here are the correct comparison numbers

Synchronous Send using Openwire Persistent Message size: 43 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 13.1 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 13.1 msgs/sec.

Synchronous Send using AMQP 1.0 Persistent Message size: 43 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 12.8 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 11.9 msgs/sec.

Async.Send using Openwire Persistent Message size: 43 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 9006 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 3678.86 msgs/sec.

Async.Send using AMQP 1.0 Persistent Message size: 43 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 21.7 msgs/sec.

Persistent Message size: 1580 byes, no compression, 200 concurrent connections, 100,000 message count, throughput around 21.2 msgs/sec.

Note:

Even though with Async.Send message delivery is not guaranteed, using Openwire I always received all 100,0000 messages while with AMQP 1.0 around 25% of messages went missing.

AMQP 1.0 Async.Send numbers are still not close to Openwire Async.Send numbers. Any other suggestion(s) ?

Thanks for your help.

Answer 1

There's not a whole lot you can do on the Broker side here to improve AMQP performance at the moment. Keep in mind that you aren't doing a true comparison here as you have disabled the sync sends on the ActiveMQ client which results in less than guaranteed delivery for your producers. If you want to try to compare them this way then you should try and make the QPid JMS client also send it's messages this way. There is a sync publish option on the QPid JMS ConnectionFactory you can try to see if you can get it to not send the messages unsettled as that will basically require an ack for each message.

I don't believe that the QPid JMS client is implemented as the most performant AMQP 1.0 client but instead is more of a proof of concept so this could change as new clients are written. Another bottleneck here is the current implementation of Proton-J which ActiveMQ uses is not very performant compared to our long hardened OpenWire transport so until that library gets better the performance will suffer. Proton-J is evolving all the time so things should improve over time.

If you are sending and receiving using AMQP client's only you can configure the AMQP transport in ActiveMQ to use the raw transformer option which would lower the work done for each message but it won't save you a whole lot.

<transportConnector name="amqp" uri="amqp://localhost:5672?transport.transformer=raw"/>

Your best bet is to dig into the QPid JMS client code to see if you can configure it to send persistent messages settled.

At this stage OpenWire is going to always be faster on ActiveMQ than AMQP so unless you have some compelling reason to use AMQP just stick to the native OpenWire client. You can try the trunk version of ActiveMQ (v5.11-SNAPSHOT) which has some additional work done on the AMQP side which does improve things a bit more and it uses the latest release of Proton-J which has some improvements as well. And this will be an area of continued work so you can expect that the performance picture will improve over time.

Answer 2

The Qpid 0.28 AMQP 1.0 JMS client sends persistent messages synchronously by default, and non-persistent messages asynchronously by default. You can make the send asynchronous as follows: adding sync-publish=false to an individual connection URL, or setting the Java system property "qpid.sync_publish" true. See https://issues.apache.org/jira/browse/QPID-5574 for details.

As Tim already mentioned, sending persistent messages asynchronously is a bit unusual (at least while not also using transactions to impose some level of synchronous confirmation for bounding purposes).