Performance Testing
Performance testing is one of those phrases on which there is little agreement as to what it precisely means. However the various descriptions given for performance testing show that there is a broad agreement about the non-functional requirements that need to be tested.
Another complication factor is that the various types of performance testing are all done using a load testing tool. The tool, like all tools, enables somebody to do a job, not to do the testing itself. For this you need a trained and experienced performance or load tester.
Performance Testing and Non-Functional Requirements
The performance based non-functional requirements to be tested cover a number of questions which the owners of the system can reasonable ask. These include:
- How many users can the system deal with?
- What performance will each type of transaction have?
- What happens if very high usage occurs?
- What happens if the system fails?
- Can the system operate reliably over a period of time?
Each of these questions then has detail numbers placed against them in order to allow testing to take place.
Types of Performance Testing
For each of the questions a separate type of test can be defined. The most important confusion is between load and performance testing. Some documentation does not mention one or the other, some swap meanings which results in a general lack of clarity. In essence the two types of testing are mirror images of each other.
This article argues that although a load testing tool is used for each type of testing, the key difference between them is the Acceptance Criteria or objectives of each type of test. The types of test matching the non-functional questions are:
- Load Testing – How many users supported.
- Performance Testing – The performance of each transaction type.
- Stress Testing – Explores high but valid usage.
- Robustness Testing – Explores the effect of unavailability of resources.
- Endurance Testing – Checks long term running of the system.
To illustrate each type of testing a simple real world model will be used. Imagine our system is a road network connecting together houses with shops, schools and workplaces. Our transactions are users going from one point in the network to another, such as from the house to the workplace. More complicated transactions might be leaving the house to go to the school to drop off children and from there drive on to work. The functional testing that would have been done would checked that each type of transaction is possible, that roads exist between each of the network points for each type of transactions they want to follow. The following paragraphs will illustrate how the various types of load testing would apply to this model.
Load Testing
Load Testing means that given a performance range can a varying numbers of users be dealt with. This is dealing with the issues of checking various patterns of load and seeing if a pattern of usage is delivered rather than checking the details of the individual transactions.
In terms of the road network model this would mean simulating a wide variety of patterns of usage. Can the school and work rush peak load perform acceptably. Can picking up children during school homeward rush be dealt with. Can delivery vans get to shops in time and not block the roads. These are all issues to do with checking the volume and mix of transactions.
Performance Testing
Performance Testing itself checks that for a given range of users loads are each of the specific transaction performance criteria met. This is involved in seeing where the problems are with individual transactions, rather than the delivery of the overall load.
The road network would show this by tracking individual users during the various road patterns and see if they get to school or work on time. Whether they are held up in traffic jams, and where the problems occur so that changes can be made to the road network.
Stress Testing
Stress Testing explores the issues of what happens to the system as it is put under valid load in excess of it design parameters. It is sometimes defined as to break the system but this is not the point of the test, though it may be the result. The purpose is to put the system under a number of very high load patterns and observe what happens to it. Some of the observations would be if the system just slows down, come to a stop, or causes serious damage.
For a road network this would be the equivalent of putting in a lot more vehicles. Perhaps the school is having an exchange visit, there is a conference in town, or the shops are having a special sale which attracts out of town visitors. Each or several of these would put a stress on the network.
Robustness Testing
Robustness Testing in contrast to Stress Testing deals with the response of the system under various loads when resources, such as a dependent system, hardware, or network, become unavailable. Again the purpose is not to break the system but study the system’s response to the various resource failures against several load patterns.
The equivalent for the road network would be having roads collapse, schools or works places shut down, or the shops deciding not the open. The purpose is to see if everybody could still get around town.
Endurance Testing
Endurance Testing checks how stable the system is over a period of time. It applies a high but normal load pattern of transactions to the system for a continuous period. Checks are made during the run to see if any subtle errors build up over a period of time which can affect the performance or integrity of the system.
In road network terms this would be checking usage over several consecutive days to see what happens. It may be that vehicles break down and stop people getting around, or because of extended jams schools and shops do not open on time. These errors build up over a period of time.
Load Testing Tools
All these tests use a load testing tool to assist in running of the tests. The Acceptance Criteria and tests have to be designed by the testers, and then implemented in the tool. The tool itself has three key functions:
- Provide a simulated load to the system which can vary with time.
- Record the results of running test loads.
- Provide tools to display the results of tests.
In road terms they would be providing more vehicles to go on runs within the network, having watchers to count vehicles and record what is going on, and then writing a report on what is found.
When to Performance Test
- Types of Testing employed during a project including when performance testing takes place.
- User Requirements - three areas to include, such as non-functional requirements, and one to avoid when writing user requirements.