The Seven Principles of Software Testing

Harshani Perera, 18 July 2018

Principles are rules or values that have been proven over time that provide a foundation for how something works or occurs. Similar to the Principles of Physics or the Principles of Law, Software Testing also consists of principles that have stood the test of time over the past 40 years. I decided to write a blog on these principles due to the valuable guidance it has provided me in my testing career and continue to do so.

I came across these principles during my university studies and the ISTQB Exam from the book “Foundations of Software Testing” by Dorothy Graham, Erik van Veenendaal, Isabel Evans and Rex Black. Looking back on my testing experience, I can now see how these principles provide guidance when testing. I always keep these guidelines in the back of my mind when testing as it helps to carry out the testing in an effective and efficient manner.

I wish I could have known these principles earlier when I first started testing. Hopefully, this blog will help my fellow testers in their path to mastering the art of testing.

So regardless of if you are just starting in your testing career or you are an experienced tester, these principles can be applied for all testing situations.

(Graham, Veenendaal, Evans, & Black, 2009)

Principle 1: Testing Shows Presence of Defects

The purpose of testing is to discover bugs in software but no matter how much testing we carry out or if no bugs are discovered, we cannot say that there are no defects in the system. Testing will reduce the probability of any hidden bugs remaining in the software however it is not a proof that the software is bug free.

Principle 2: Exhaustive Testing is Impossible

As a tester, one of the most important concepts to be aware of is that it’s not possible to test everything i.e. testing all the possibilities, although you might be tempted to. When testing, you have to keep in mind that software development projects work on budgets and constraints. Testing all the scenarios will require more resources therefore will require more time and more money. This would lead to going over budget and over time. This in turn will cause project failure.

To combat this, we need to determine where to focus our testing effort based on risks and priorities. Testing should also provide good coverage of the entire project rather than focusing all the testing effort in one specific area. Create a test plan according to the risks, priorities and coverage given the budget and time. This will lead to more effective and higher quality testing.

Principle 3: Early Testing

The earlier the bugs are found, the less the cost to fix them. Bugs found later in the development stage are expensive to fix. The following graph shows the cost of the project compared to the time the bugs are found.

Figure 1: The rising costs of finding and fixing defects. (Ambler, 2002)

Therefore, it is best to start testing as early as possible in the software development lifecycle (SDLC). Conduct early test design, requirements and reviews to capture any gaps and fix them prior to starting development work. It is a good idea to involve the test team right from the beginning of the SDLC so they have a good understanding of the product and can create more effective tests.

Principle 4: Defect Clustering

This means that defects are not distributed evenly across an application and tend to concentrate or cluster in a few areas. Thus, a few modules of the application will contain the most number of defects.  We can apply the Pareto Principle,  (the 80-20 rule) which says that 80% of the bugs are in 20% of the code or modules.

This information can be used during risk assessment for test planning to allocate more time for testing areas with known complex code/sections of the application. Testing these complex areas early can help to reduce the time and cost of finding defects.

Also if you come across a defect during testing, it is likely there will be more in that area. The cause of these clusters is because some areas of the code can be complex and tricky or making changes can have a rippling effect in the software and create more defects.

Principle 5: Pesticide Paradox

This principle says that if the same set of test cases are repeated over and over again, it will no longer find any new bugs. This is similar to insects building up resistance to pesticides over time. The reason for this is when bugs are initially found and fixed, the number of bugs will reduce as the software has improved hence the same tests will not find any new bugs as the defect clusters change over time.
To prevent pesticide paradox, testing now needs to focus on other areas or risks such as looking at the requirements and design documents for defects, writing new test cases to test different areas of the system that were not tested previously and writing test cases to test different paths or using different test data.

Writing more test cases and testing them will require more testing time hence increase the cost. Therefore it is important to work according to the priority of test cases. Review and update your test cases depending on changes to the product, new releases and remove any unwanted tests. This will help in managing the testing time.

As the 'hot spots' for bugs get cleared up, we need to move our focus elsewhere, to the next set of risks. Over time, our focus may change from finding coding bugs, to looking at the requirements and design document defects, to looking for process improvements to minimise bugs in the software.

Principle 6: Testing is Context Dependent

Testing needs to be tailored according to the type of project. For example, testing an air traffic control system should be different to testing an online shopping website.
Different software systems have different levels of risk and impact therefore testing needs to reflect this. More focus and more rigorous testing will need to be carried out for an air traffic control system than the online shopping website because it has a much higher risk and impact if defects were found in the software compared with the online website.

In some projects, a spelling error can seem trivial. However for another project this could have a major impact on the reputation of the business. Hence, when presented with a project, you should first recognise the risks and impact of the project in order to determine the different methodologies, techniques and the types of testing to apply. This will help to provide a quality product and manage the testing time and budget.

Principle 7: Absence of Error Fallacy

It may seem that the software has been tested and all the defects have been fixed.. However the system could still end up being unusable by the client. This can occur when the software developed did not meet the business needs and requirements of the user.

The main purpose of software is to assist the user in completing their tasks effectively and efficiently. Hence when conducting testing activities, it is important to keep the user needs and requirements in mind. Test cases should not be developed for the wrong requirements.

It’s important to have reviews and meetings to discuss the requirements with the whole team.

It may seem like these principles are just theory and will not really be used in practice when carrying out testing work. However this is not the case. Keeping these principles in mind will help to plan testing activities and test software diligently in a methodical manner that can save time and money. If the principles seem too daunting at first, you can start applying one principle at a time and slowly work through the rest. Through experience, you will automatically apply these principles in your thought processes when testing.

References

Ambler, S. (2002). Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process. New York: John Wiley & Sons, Inc.

Graham, D., Veenendaal, E. v., Evans, I., & Black, R. (2009). Foundations of Software Testing ISTQB Certification. Hampshire: Cengage Learning EMEA.