Copyright © 2011 Corvus International Inc. All Rights Reserved
Success is often
achieved by those
who don't know that failure is inevitable.
Coco Chanel (1883 - 1971)
Too Many; Too Few
There are two situations that scare testers:
Testing and the Five Orders of Ignorance
If we look at the Five Orders of Ignorance we can see that we mostly test for two of the Orders:
- Zero Order Ignorance (0OI): I have 0OI when I provably know something
Testing for 0OI is asserting the provably part of the definition of 0OI. It is the "clean" testing where we ensure that the knowledge we put into the system is, in fact, correct.
This kind of testing is finite since there are only a certain number of things we want a system to to.
The ideal result from this kind of testing is 100% passing--we prove that the knowledge is correct.
- Second Order Ignorance (2OI): I have 2OI when I don't know that I don't know something
Testing for 2OI is much more difficult. We are looking for something but, by definition, we don't know exactly what it is. We are looking for something that increases our awareness of our ignorance.
This kind of testing is essentially infinite, since there are any number of things a system might do.
This kind of testing increases our knowledge by allowing us to expose our 2OI and then resolve that ignorance by uncovering the related knowledge; but only if the test "fails" (that is, throws a defect). Therefore a certain number of exposed defects is actually valuable.
We should never test for:
- First Order Ignorance (1OI): I have 1OI when I don't know something (but I am aware that I don't know it)
Testing for 1OI would mean that we already knew that there was a problem in the system, but chose to run a test rather than fix it first. In practice, we would would fix the lack of knowledge first and then run a 0OI test to prove we'd fixed it.
We so also run a certain amount of tests for:
- Third Order Ignorance (3OI): I have 3OI when I don't know a suitably efficient process to find out that I don not know that I do not know something
We usually do run tests to prove that the test system works as expected and required. These tests are proving the suitably efficient part of the definition of 3OI. However, this is not the focus of testing--we don't run testing to show that testing works.
Not too Hot, Not too Cold; Not too Big, Not too Small
There is a "Goldilocks" number of defects where we are seeing the optimal number of errors. Where would this point be? If we view a thrown error as carrying information from the system to the tester, at what point is this information content maximized?
Back in 1982, a telegraphic engineer by the name of Ralph Hartley published a formula that showed the maximum point at which information would be transferred in a telegraph message. Viewing the test result as a telegraph message from the system, we can apply this formula to testing. The formula is:
Information Content = Pf * log [1/Pf ] + Ps * log [ 1/Ps ]
Pf = Probability of failure and
Ps = Probability of Success
The graph of this equation is shown here.
The Goldilocks Point
This means that the "Goldilocks" point in testing is at 50%. When we design tests such that half of them "succeed" (do not throw an error) and half of them "fail" (do throw an error), we maximize what we learn from testing.