24 WHEN CAN WE STOP TESTING?

24 WHEN CAN WE STOP TESTING?

2015-09-25

THERE IS NO simple way of deciding when a system is completely tested. Most authors agree that there is no single criterion we can use in order to decide that we have fi nished the job. Here are some variants of opinion.

24.1 Five Criteria for Completion of Testing

According to Lee Copeland, there are five elementary criteria which, together, are usually used for decided when you can stop testing. 109 These are when:

1. We have achieved the coverage aims we defined in the strategy
2. The number defects discovered is lower than the boundary value we defi ned
3. The cost of detecting more defects is larger than the estimated loss arising from remaining defects.
4. The project team draws the collective conclusion that the product is ready to be released.
5. The decision maker gives the order to go to production.

There is a multitude of weaknesses in these criteria when taken one at a time:

1. The aim of reaching a certain level of coverage may run the risk of driving the testers towards writing fewer or inferior test cases, simply in order to manage running everything.
2. Not discovering any more defects may be down to us not testing in the right way, or the best testers being on holiday, but it does not necessarily mean that there are no more defects.
3. The cost being more than the benefi t of more test cases is a highly subjective evaluation which the tester is certainly not qualifi ed to make: this is more a business issue.
4. The consensus of the team may be a relatively better measure since, here, we discuss matters with developers, testers and people familiar with the operation together.
5. The last criterion is a deadline which is decided outside the testers’ domain, and actually has nothing to do with how well we have tested, but rather builds on a pure business judgement that we have to release the product by a certain date.

24.2 Good-Enough Quality

James Bach describes an approach he calls Good-Enough Quality, which he summarises with the following four points which must all be fulfilled:

1. It has enough benefits
2. It has no critical problems
3. The benefits suffi ciently outweigh the drawbacks
4. In the situation at hand, with all things considered, further testing and improvements would do more harm than good.

To arrive at the above, we can explain each part in a little more detail. Ask the following questions:

1. Which specific advantages does our product have? How great is the likelihood that one of our target customers will make use of a particular advantage? How important is each advantage? Which parts are critical? Are all advantages good enough for our target customers when taken together?
2. What potential problems does our product have? How great is the likelihood that one of our target customers will be exposed to a particular problem? How damaging can each problem be? What problems are wholly unacceptable? Are all problems, when considered together, too many for our target customers to be satisfied?
3. Does the product have enough advantages so that the drawbacks that happen to arise are few enough? How good does the product have to be in order to be accepted by the customers?
4. In what ways could we improve the product? What would this mean for our costs? Is there the possibility of delivering now, and then delivering improvements later? Precisely what advantages would