Amazon Web Services application: Performance testing using Loadrunner 12.02

Question

I have been asked to do a proof of concept for performance testing an application which our organization will be hosting on Amazon Web Services. The application will be used by around 460 people. If only an end-to-end test of the whole application is needed, we will set up tests from the users' perspective without measuring the performance of any individual component. If the web service server needs to be performance tested at an integration level, it is an interesting challenge because we have not tested cloud-based web services before. I did a search on this website and it gave me results for using JMeter to do a similar task. While this is an option, we use LoadRunner 12.02 here and I am looking for advice on how to use/get started with that. So far, I have found that LR 12.02 integrates with Amazon EC2, and that we need to create a custom image in the cloud. If anyone has tested AWS using Load Runner, any information would be useful: where to start, possible issues, config issues/resolutions, running tests locally VS on the cloud etc. Feel free to ask me for more details if you need.

Look forward to your suggestions. Thanks in advance!

Answer 1

Testing of AWS based applications with LoadRunner happens all the time. One thing to be aware of is that you will want to run your load generators inside of AWS simply because you get charged for bits sent into/out of the cloud, but not for bits generated and consumed inside of the Amazon data center.

You are running in a VM environment. Your load generators will be on VMs. Your application will be on VMs. This has implications for test repeatability related to environment initial conditions on shared resources arbitrated by the hypervisor of the VMs. This also applies to in-test conditions. In short, you cannot control what is going on with the other VMs that may hit an arbitrated resource. This will push your average, maximum and standard deviation higher. This is true for any test tool. You also have a clock float issue to content with as the load generator clock will float related to the physical clock. at times of clock synch with the underlying hardware you will have a clock jump ahead to the synch point. This will result in artificially longer timing records, which accounts for the other half of the longer overall timing records when combined with the uncontrolled initial and in-test conditions.

Installing load generator software in AWS is no different than installing on any other host. Just remember to shut down your load generators at the end of the test to avoid extra costs for running VMs that you aren't using.