Articles...Copyright © 2011 Corvus International Inc. All Rights Reserved

*Success is often
achieved by those
who don't know that failure is inevitable. *

**Coco Chanel (1883 - 1971)
French Fashion Designer**

Testing: failing to succeed

Phillip G. Armour

Communications of the ACM, 2011

Communications of the ACM, 2011

**Too Many; Too Few
**

There are two situations that scare testers:

Too Many DefectsThis usually indicates the system is not fulfilling its requirements, is poorly constructed,or has other limitations- T
oo Few DefectsThis might mean the system is really good, but it more often means that the testing process is not working

**
Testing
and the Five Orders of Ignorance
**

If we look at the

we can see that we mostly test for two of the Orders:Five Orders of Ignorance

Zero Order Ignorance (0OI): I have 0OI when ITesting for 0OI is asserting theknow somethingprovably

part of the definition of 0OI. It is the "clean" testing where we ensure that the knowledge we put into the system is, in fact, correct.provably

This kind of testing is finite since there are only a certain number of things we want a system to to.

The ideal result from this kind of testing is 100% passing--we prove that the knowledge is correct.

Second Order Ignorance (2OI): I have 2OI when I don't know that I don't know somethingTesting for 2OI is much more difficult. We are looking for something but, by definition, we don't know exactly what it is. We are looking for something that increases our awareness of our ignorance.

This kind of testing is essentially infinite, since there are any number of things a systemdo.might

This kind of testing increases our knowledge by allowing us to expose our 2OI and then resolve that ignorance by uncovering the related knowledge; butif the test "fails" (that is, throws a defect). Therefore a certain number of exposed defects is actually valuable.only

We should never test for:

First Order Ignorance (1OI): I have 1OI when I don't know something (but I am aware that I don't know it)Testing for 1OI would mean that we already

that there was a problem in the system, but chose to run a test rather than fix it first. In practice, we would would fix the lack of knowledge first and then run a 0OI test to prove we'd fixed it.knew

We so also run a certain amount of tests for:

Third Order Ignorance (3OI): I have 3OI when I don't know aprocess to find out that I don not know that I do not know somethingsuitably efficient

We usually do run tests to prove that thetestworks as expected and required. These tests are proving thesystempart of the definition of 3OI. However, this is not the focus of testing--we don't run testing to show that testing works.suitably efficient

__
Not too Hot, Not too Cold; Not too Big, Not too Small
__

There is a "Goldilocks" number of defects where we are seeing the optimal number of errors. Where would this point be? If we view a thrown error as carrying information from the system to the tester, at what point is this information content maximized?

Back in 1982, a telegraphic engineer by the name of Ralph Hartley published a formula that showed the maximum point at which information would be transferred in a telegraph message. Viewing the test result as a telegraph message from the system, we can apply this formula to testing. The formula is:

Information Content = Pf * log [1/Pf ] + Ps * log [ 1/Ps ]where

= Probability of failure and:

Pf= Probability of Success

PsThe graph of this equation is shown here.

**
The
Goldilocks Point
**

This means that the "Goldilocks" point in testing is at 50%. When we design tests such that half of them "succeed" (do not throw an error) and half of them "fail" (do throw an error), we maximize what we learn from testing.