Copyright 2011 Corvus International Inc.  All Rights Reserved

Home      About Corvus       Contact Us       Articles/Resources       Clients          Affiliations  

Success is often achieved by those
who don't know that failure is inevitable.

                                            Coco Chanel (1883 - 1971)
                                 French Fashion Designer

ACM DL Author-ize serviceTesting: failing to succeed
Phillip G. Armour
Communications of the ACM, 2011


Too Many; Too Few                                              

There are two situations that scare testers:

Testing and the Five Orders of Ignorance           

If we look at the Five Orders of Ignorance we can see that we mostly test for two of the Orders:

We should never test for:

We so also run a certain amount of tests for:

Not too Hot, Not too Cold; Not too Big, Not too Small           

There is a "Goldilocks" number of defects where we are seeing the optimal number of errors.  Where would this point be?  If we view a thrown error as carrying information from the system to the tester, at what point is this information content maximized?


Back in 1982, a telegraphic engineer by the name of Ralph Hartley published a formula that showed the maximum point at which information would be transferred in a telegraph message.  Viewing the test result as a telegraph message from the system, we can apply this formula to testing.  The formula is:

Information Content = Pf * log [1/Pf ] + Ps * log [ 1/Ps ]

= Probability of failure and
= Probability of Success                                                  

The graph of this equation is shown here.



The Goldilocks Point                                                   

This means that the "Goldilocks" point in testing is at 50%.  When we design tests such that half of them "succeed" (do not throw an error) and half of them "fail" (do throw an error), we maximize what we learn from testing.


Testing: Failing to Succeed