REST API Response time increases under load [Tomcat]

Question

We have one REST API (GET) which can be called by lot of mobile users at the same time. Our current user base is around 300k but is expected to grow around 1 million.

The API is simple. It makes 3 parallel requests using Akka, and returns the combined result. The main code looks like this:

        Future<List<CardDTO>> pnrFuture = null;
        Future<List<CardDTO>> newsFuture = null;

        ExecutionContext ec = ExecutionContexts.fromExecutorService(executor);

        final List<CardDTO> combinedDTOs = new ArrayList<CardDTO>();

        // Array list of futures
        List<Future<List<CardDTO>>> futures = new ArrayList<Future<List<CardDTO>>>();

        futures.add(future(new PNRFuture(pnrService, userId), ec));
        futures.add(future(new NewsFuture(newsService, userId), ec));
        futures.add(future(new SettingsFuture(userPreferenceManager, userId), ec));

        Future<Iterable<List<CardDTO>>> futuresSequence = sequence(futures, ec);

        // combine the cards
        Future<List<CardDTO>> futureSum =  futuresSequence.map(
                new Mapper<Iterable<List<CardDTO>>, List<CardDTO>>() {
                    @Override
                    public List<CardDTO> apply(Iterable<List<CardDTO>> allDTOs) {

                        for (List<CardDTO> cardDTOs : allDTOs) {

                            if(cardDTOs!=null)
                                combinedDTOs.addAll(cardDTOs);

                        }

                        Collections.sort(combinedDTOs);

                        return combinedDTOs;
                    }
                }
        );

        Await.result(futureSum, Duration.Inf());

        return combinedDTOs;

The 3 futures are simple select statements from a MY SQL database which execute under a milliseconds. We are using Spring + Hibernate here.

The whole API takes 50ms to return the result on an average.

Now, while we were doing the performance testing with 3 servers, we came to conclusion that after around 200 requests / seconds, the response time of API starts increasing linearly. It goes as high as 3 -5 seconds under load. The surprising part is that CPU usage revolves around 20% at that time and there is nothing significant going on in JVM Memory. The memory usage is around 700 MB. We have 16 GB

I am unable to find where the bottleneck is. How can I scale this API to at least 1000 requests / sec. I am at least looking for pointers on where to start. I have explored tools like top , visualvm, but do not find anything alarming.

This our JVM settings on Java 7

export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -server -Xms4g -Xmx16g -XX:MaxPermSize=1g -XX:PermSize=512m -XX:MaxNewSize=4g -XX:NewSize=512m -XX:SurvivorRatio=16 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=12 -XX:LargePageSizeInBytes=256m -Dspring.profiles.active=staging -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9899 -Djava.rmi.server.hostname=$HOSTNAME -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

I have read these questions and it seems that its a general trend. Would switching to some other framework like node.js or Erlang help?

Response time Increases as concurrency increases in Java

Tomcat response time is increasing as concurrency is increased in apache bench

The connectors default maxThreads value is 200. Take a look at tomcat.apache.org/tomcat-7.0-doc/config/http.html — user
I have tried increasing that value to 5000 with BIO connector — Madhur Ahuja
Be careful with Await.result(futureSum, Duration.Inf()); statement. It blocks the thread so may create performance problems. — Mustafa Simav

Adam Gent Adam Gent · Accepted Answer · 2015-11-29T18:26:37

Its impossible to know for sure where your performance issue is but I have found its generally because of (given your description):

A thread or connection pool that is having contention issues (either database connection pool or tomcat request pool)
A synchronized variable/code or BlockingQueue (which can be a superset of the above).
Bad load balancer or configuration
Bad network

What I recommend you do is to isolate as much as possible. First prove it is not the database connection pool. That is run the same concurrent load but only doing the database portion. Remember three servers requires 3x connections.

Next run a 1-3 servers making mock responses with out doing any data processing with and w/o the load balancer. You would be surprised how often the load balancer / network can cause issues.

Continue separating out things.. test, observe, repeat.

Finally when you have isolated it really to be Tomcat you may want to read what Netflix does.

REST API Response time increases under load [Tomcat]

1 Answers