Web Server Performance Degradation

0
votes

The web application is running on Springboot and deployed on WebLogic. We have assigned 400 as max threads and JDBC to be 100 connections.

When we perform load testing on the web application, the performance is optimal when the load is low (the response time is less than 200ms for most of the http request that we called).

When we increase the load, we can see that the thread count increases and jdbc count also increases gradually but no where near to max. However, the response time is getting much longer and it could take more than 5 seconds to response.

CPU usage, thread count, memory, JDBC connection seems to be normal during these period.

Another observation is that during testing and we saw that the performance is degrading, we used another machine to make a http call to the server that is only retrieving text without any DB or logic, and even this simple http call will take 10s to respond. (And the server resources is still not MAX!)

So, we are wondering what keep them waiting ? Any other possible bottleneck?

2
spring-bootperformanceserverjmeterweblogic
400 threads and 100 db connections !! How many requests per second are you expecting ??? First action is to take several thread dumps when performance is degrading spaced 3 seconds apart. Then, analyze your thread dumps to see where the java code spends time. - Emmanuel Collin

2 Answers

0
votes

If the server doesn't lack resources like CPU/RAM/etc. only a profiler can tell you where your application spends the most time which might be in:

  1. Waiting in a queue for next thread/db connection from the pool to be available
  2. Slow database query
  3. Inefficient functions/algorithms which a subject to optimization
  4. WebLogic configuration not suitable for high loads
  5. JVM configuration not suitable for high loads (i.e. system is doing garbage collection to often/too long)

So I would recommend re-running your test with profiler tool telemetry enabled and at the same time monitoring essential JVM metrics using i.e. JMXMon Sample Collector which can be used for monitoring your application-specific metrics as well. It's a plugin which can be installed using JMeter Plugins Manager

0
votes

For a detailed approach on how ago about identifying poor thread performance I suggest you take look at the TSA Method by Brendan Gregg.