JMeter how to build a reliable load/stress test

Question

I’ve been looking for a good resource that gave an idea of how to create a reliable load / stress test, but I haven’t been able to find anything.

Let’s say that I want test an application that is used to sell books.

It has been proved that the day with the highest sales is Christmas Eve (December 24th) with a maximum peak of 21 sales per minute. Taking a closer look to these results, it was determined that there was a maximum peak of 2 sales per second like it’s shown in the image below.

So taking into account all the results mentioned above, this is what I've tried so far:

Load Test:

• Number of threads: 2

• Ramp-up period: 1 second

• Loop count: forever

• Duration: 3600 seconds (1 hour)

Stress Test:

• Number of threads: 4 (an increase of a 100%)

• Ramp-up period: 1 second

• Loop count: forever

• Duration: 3600 seconds (1 hour)

Now that you know what I've tried, I want to ask you some questions:

1. How you create a load and stress test so that they’re reliable? Which things do you consider?
2. Is it correct to increase the number of threads, but keep the ramp-up period invariable?
3. Do you think it’s OK to use a duration of an hour?
4. For my stress test, I increased the number of thread in a 100%, what do you think about this? Should I use a higher value or perhaps an increase of a 100% is too much?

Thanks in advance.

Answer 1

uhmm, rather complex question...

I usually start writing down a scenario, that is, a sequence of action that is supposed to be done by a user. In your case, the final action is the book buying, but probably there is much more to be done before, like browsing the catalog, do some search, choosing a title, register personal data, etc.

It can be very useful to go deeper with site statistic analysis.

Once you have a script working, you can set the timer so that they emulate the user behaviour.

So you will go finding that this script take much more that 1 seconds. Probably, in the real world user spend at least 5 minutes on a site before buying a book, but again check these data with the marketing people.

Let say that each loop take 5 minutes, this gives 300 s. If you want to reach 1 sales/s you need 300 threads to get there.

If you start the threads all together, you will test a very unrealistic and pessimistic situation, where all the 300 user will buy the same book at the same time. So you have to randomize everything: searches, the book list, the timers, so that each user seems really independent and different from the other.

Once you have done that, you have done point 1.

2) you can use rampup time just control the time it takes to launch all the thread. 5 threads /second is a safe figure in my experience.

3) again it depends on the user modeling you are building: if the basic script takes 5 minutes, even 30-40 minutes seems a save duration to me, but YMMV.

4) this is pretty much depending on target your test has. Try to clarify this with your customer. I usually do test with 60% 80% 100% and 120% of the target load, but each situation is different. Also, take in account the response time target and error rate target: it makes no sense to go with upper load if the lower one fails the target.