MDB + ActiveMQ + Weblogic with high load

0
votes

I have a company project where we are using EJB (MDB) + ActiveMQ (connected with weblogic using activemq-rar connector) + Weblogic 12 for processing transactions using QUEUEs. We have Oracle RAC as our database Our module is based on 3 MDBs:

  1. The first MDB receives messages from external resources using a queue in ActiveMQ, and puts these messages in other queue.
  2. The second MDB receives the messages from #1, stores in the database, and puts the message to another queue.
  3. The last MDB receives the messages from #2, applies business rules, stores in database, and routes the message (depending of some routine rules) to different topics.

We need to archive more than 700 messages/second. We have JMeter to test the load.

We did a lot of tuning actions in Weblogic and ActiveMQ and a lot of fixes in the code but we only can process 450 messages\second. With more load the messages start to accumulate in the last queue (step #3 above). The IT platform (Weblogic, ActiveMQ, and database) is enough. It is not the problem. We have a consumption lower than 30% in CPU\RAM with our load of 450 messages\second.

We played with a lot of ActiveMQ and Weblogic configurations:

  1. Weblogic:

    1. Self Tuning Thread Minimum Pool Size -> 0, 10, 100...200 threads
    2. Self Tuning Thread Maximum Pool Size -> 100, 200, 800...1500 threads
    3. JDBC (Oracle GridLink) with Initial Capacity (connections) of 100 and Max Capacity of 400,600..800

  2. MDBs:

    1. Each MDB has initial-beans-in-free-pool = 10, 50, 100...200... in the weblogic-ejb-jar.xml
    2. Each MDB has maxSessions = 100, 200, 400... activationProperty (for ActiveMQ) in the ejb-jar.xml

  3. ActiveMQ:

    1. Openwire configured with more than 1000, 2000...5000 connections
    2. Initial capacity (10, 50...100) and Max capacity(100,600...1000) in the ActiveMQ pool configured using ActiveMQ-rar (weblogic-ra.xml).

We are using XA transactions (AMQXAConnectionFactory)

It does not matter how we increase all these values, the results are always the same. We inspected the software with VisualVM (JMX) and Weblogic does not assign more threads to process messages although a lot of threads are totally idle. We think that something is limiting our performance, but we don't know what it is.

We played with all these values (increasing, decreasing....), but we cannot improve our best results (450 message/second). We are totally lost. We also have tried with Workmanagers for our server but we got the same results. Could you help me? Any idea?

1
performanceweblogicactivemqweblogic12cmessage-driven-bean
Yes, all the steps have persistent JMS messages and XA with the database and ActiveMq – Javier Sanz Hernández
Are you suggesting these to try to check if the problem is related with the persistence or XA? – Javier Sanz Hernández
No. The Disk is not the Problem. Our IT Team checked and everything is correct – Javier Sanz Hernández
What exactly do you mean by "everything is correct"? The issue isn't really about correctness. It's about speed. Even fast disks can be a bottleneck if syncs pile up. – Justin Bertram

1 Answers

0
votes

If you're using persistent messages and XA transactions at every step then the bottleneck is likely your disk(s). Acknowledging a persistent message requires a sync to disk. Sending a persistent message requires a sync to disk. Updating the XA transaction log likely requires 2 syncs to disk. Obviously persisting data to a database requires a sync to disk. You're doing many of these on each step which will add up to a lot of disk syncs.