Differentiating between load, performance and stress testing

[Ronan Healy]

I have had a discussion recently with a few testers and have seen it mentioned on some blogs too about using the terms load, performance and stress testing. I have seen some testers use the terms interchangeably but that they are not the same.

I know that there are differences between the three and that these differences can be subtle. How would you define the difference between the three? Do these difference matter?

[Mark]

Hi,

ISTQB put it as:
Volume Testing = Large amounts of data
Load Testing = Large amount of users
Stress Testing = Too many users, too much data, too little time and too little room

For me performance testing covers all the areas, it is about how the system performs. Loadtesting is about how the system performs under a specific load and stress testing is about looking at how a system performs under a larger load that stress it resources.

I don’t think any of these terms are really particularly helpful, if as testing professionals we are having this debate how confusing is it for the customer.
I stay away from these terms and instead talk about an average day average hour performance test, or a peak day peak hour performance test etc. These at least identify what the test is really about.

It is important that we clearly identify what we need to test and why. Loading a system up until it fails is fine, but unless we what load the system has to cope with we don’t know if the test is showing if the system under test is over or under performing.

I find the following names for tests much more useful and meaningful when talking to other project members and customers:

XXX loadtest – where XXX details the actual live like scenario being generated eg. ‘Peak day peak hour’, or ‘Summer Registration Peak’. It should be something the customer can understand. These test demonstrate the system can cope with expected volumes.

Soak Test or Endurance test – This is a test that runs a reasonable load for an extended period of time to identify issues with resource leaking, or logs filling etc.

Capacity/ Headroom/ Step test – This runs the load up to peak level and then increases the load in stages allowing the system to stabilise for 10 minutes between each increment. Generally I would run this until the resource utilisation or response times became excessive or impractical. I would not normally run it to destruction!!

Scalability test – This is a test to see how the performance scales as the system under test is scaled horizontally and/or vertically.

Degraded Mode Test – This is a loadtest to see how the system performs in a degraded mode.

Login Rate Test – This is a test to see at what rate the system can accept logins and still provide a useful service. it is used to see if the system can cope with everyone can login at around the same time after a system or network outage.

I hope that helps.

Mark

[ramesh]

Good and healthy discussions
1. Performance testing is just 30-40% of performance engineering. It is defined as the testing activity carried out to understand the behaviour of the application when subjected to load.
2. Rest all types of testing like load, stress, scalability, endurance/reliability, volume, spike are just different types under the performance testing exercise.
3. Load-testing is defined as the performance testing executed to understand the behaviour of the application at low, normal and abnormal load
4. Endurance / reliability / long- duration testing is conducted where the application is subjected to a defined load for specific duration of time that can vary anywhere between 2 hours to 40+ hours under steady load the outcome shall be used to find out and memory leaks both from application and server point and associated other parameters
5. Stress testing is testing activity under performance testing umbrella to understand the load at which the application starts to degrade. (Note: the objective is not to break the system, this is a misunderstanding in the non functional testing community.)

[Alin]

Hi Ronan,

I see a connection between these types of testing because you can test something using these entire 3 techniques one after each other. From my point of view, they are like this:
– Performance testing: for example, in a web application performance testing is related to the response time like how much time it takes to open the page, load the content, run a login process, send/receive data to/from database etc. On a desktop application it could be used to see how long it takes to be installed, how fast the application is opened/closed and the menus/pages are displayed etc.
– Load testing is focused more on the repetitive actions that might cause a failure of the system and it is about large volume of data and interactions with the system. For a web application it checks what happens if there are many users (hundreds, thousands etc.) that simultaneously perform an action like a login. It is important to see how the system handles many HTTP requests or large volumes of data that are uploaded or downloaded. On a desktop application It can test what happens if the application is installed and uninstalled many times, what happens if you open it several times after it is opened, how many instances of the app can be run simultaneously without filling up the system’s memory with temp files etc.
– Stress testing reveals the behavior of the system when it runs outside the limits that were specified for that system to work properly. If a web application can handle maximum 1000 users, what happens if 2000 users make a login: will the system crash, will there be 1000 logins successfully performed, will the system go very slow etc.? If it crashes, can it be recovered automatically and how long does it takes. On a desktop application it can test what happens if the system is stopped while installing the app: can it be further installed or it can damage something in the registry area of the system? Stress testing might be like driving at 200 km/h a car that has construction limit of 180km/h: will its engine stop, will a message displayed on dashboard, or simply nothing ?

In practice, it could be a very small difference when applying these techniques. If the testing process requires, all of them might be applied.

Regards,
Alin

[Albert]

@Rameshviyer, I agree with you and I really like the reference you make to performance engineering. Although they don’t have to go together, to me performance testing starts to get useful when part of performance engineering. Without it being part of performance engineering it turns into just validating requirements. And although that has its value, there are IMO so many caveats that all too often it is waste of money.

@Alin. Strictly speaking you’re not wrong. But I don’t agree with your definitions. It remains a matter of opinion, but in general load testing is not about when an application fails. Usually that term is used for testing the performance of a system, when that system has a ‘real life’ load on the system. Best is to define the load as the expected peak load.
Stress testing is usually seen as putting load on the system to see when it breaks. Usually that is indeed outside the specified load. But in itself that is not the purpose. You want to see when it breaks. And you want to monitor the hardware resources carefully to see how the application behaves. This helps you determine the limits, how to scale if needed as well as realistic boundaries to the performance.
Again, I’m not saying that I think you are wrong, I just think Rameshes answer is more useful and much more in line with how most people define the different tests.

One more thing on endurance testing. People usually mention the obvious reason why endurance testing has value: the detection of memory leaks. Indeed a good and possibly the best example. But there is another reason. All to often components in the underlying stack tend to optimize over time. I.e. databases and bind variables. Running an endurance test early on can improve your results and give a more realistic result.

Albert Witteveen has been working both as an operations manager and a professional tester for nearly two decades now. The combination of setting up complex server environments and professional testing almost automatically led to a specialisation in performance testing.

He wrote a practical guide to load and stress testing which is available at Amazon. This books discusses how to do performance testing, how to provide real value and how to assess the performance in an objective way. It describes how t

[Alex]

In my opinion:

Stress test: applying load above expected to see how system degrades

Load test: any test when we apply multi-user load

Performance test: any test where we are interested in / measuring performance in any way

Stress test is a variation of load test, load test is a variation of performance test.

My reasoning: a single-user test measuring performance is a performance test – but I don’t think we can name it load test. I don’t see any reason why we can’t name any multi-user test load test or performance test (so for multi-user load they probably may be used interchangeably) .

I saw quite a lot of different opinions on the subject – but usually without any reasoning.

Regards,
Alex

[Alin]

@Albert
Thank you for your feedback!
Regarding the load testing, I also don’t see it about when application fails. I see it as a validation for meeting requirements related to an increasing level of load (from low to high) on real scenarios, as you said “real-life” load. The load is something expected to be and generally it can be like comparing the “actual results” when running the load tests with “expected results”.
About stress testing, I agree with you that it is performed to monitor the resources and the application’s behavior. Basically the load testing is increased over the app’s limits. But I also think that one of the purposes is to try breaking the system. In practice, I don’t think there are many specifications regarding the behavior of application outside its boundaries, so this type of testing is like an investigation made on the system which reveals some information and help the testers understand the behavior and create new scenarios.
Regards,
Alin

[QA Guards]

Performance/Load/Stress tests – these are different types of non-functional test, despite quite close to each other which makes them so confusing. To differentiate, remember of the following characteristics (which are in line with #ISTQB):
– Performance – always checks parameter(s) with regards to TIME (how much time it takes to load 200 records in the window / how many records could be saved in DB during 1 minute)
– Load – checks software behaviour while handle different (usually we think of big) volume of data, but WITHIN declared in requirements/design, meaning the volume of data we expect to be handled correctly.
– Stress – mostly the same as Load, but with data volume BEYOND declared characteristics, so we check software behaviour with data volume we do not really expect to be handled.

[Ronan Healy]

Thanks for all the replys guys. I can see why it’s easy to get confused on the differences between these types of testing. I like your idea @mark of using “average hour performance test or a peak day peak hour performance test”. It probably is more clear to the client then.

I am surprised at the number of different definitions on these tests and how the tests fit in with each other (i .e. on same level or all part of performance engineering)

I’m curious though. How did so many different definitions come about?Is there any reason for. Do testers use these tests differently?

[Alex]

@Ronan Yes, I believe that providing specific descriptions, as Mark advised, makes perfect sense. Actually, I doubt that anything else makes sense – the categories we discuss here are too broad to convey meaningful information. Yes, all these tests are parts of performance engineering and the art of performance engineering / testing (along other things) is to find out a set of tests to minimize performance risks. The reason why we have so many definition is that nobody cares – these names are rather emotional / descriptive than an exact definition of specific tests. In many cases it is basically the same test – you setup system, define load, create scripts – and then start running tests. Run for expected workload – name it a load test in your report. Applied heavier load – name it a stress test. Ran for a long time – name it endurance. Run load in increments – name it capacity test. But technically it is exactly the same test – you change, for example, only the number of virtual users and its length. And vise versa, you may ran very different tests technically – but still all of them gets in the same category (say, load testing).

@Alin “I don’t think there are many specifications regarding the behavior of application outside its boundaries” – actually it should be a must for every open system where you may have potentially unlimited number of users. If you may get more users than you system may handle, you need a way to address it (reject users, auto-scale, degrade service, or some combination of solutions). And you need a stress test to verify that it works.

@QA Guards If we test volume of data, we have another term for this – ‘volume test’. While, I guess, there are some people who will name single-user processing of different data volumes load test – I believe that it is rather confusing.

[Author]

Posts