Articles...

Copyright 2009 Corvus International Inc.  All Rights Reserved

Home      About Corvus       Contact Us       Articles/Resources       Clients          Affiliations  

A bug is a system's way of communicating with us. 

Specifically, it is how a system tells us

we are not as smart as we think we are...


                           
From a
Testing Workshop
                     Phillip G. Armour

 


ACM DL Author-ize serviceNot-defect: the mature discipline of testing

Phillip G. Armour
Communications of the ACM - Voting systems, 2004

 


 

Things have picked up somewhat recently, but a few  years ago Glenford Myers started his classic book The Art of Software Testing with a complaint about the dearth of books on the subject.  Given the gazillion books on programming languages (heck, there are around 2,000 books on Python alone), software testing still seems a little thin in the book department (and many books available on "software testing" are related to testing specific languages).

Building versus Learning                               

I have addressed this over and over (it's what I do), but the "building" of systems is really the by-product of the activity of learning what the system has to do and how it has to do it.  Interestingly, the only phase of systems development that intentionally, consciously, and deliberately seeks to find out what we don't know is testing.  The primary focus of every other phase is to build something: write the spec, approve the design, bang out the code, etc.

Two Types of Testing                                   

If you look back to The Five Orders of Ignorance, I'll focus on Zeroth through Second Order Ignorance (0OI, 1OI and 2OI).  There are two reasons we test:

  •  To find out what works

  •  To find out what doesn't work

Interestingly, we would only test for 0OI and 2OI, we would never test for 1OI.  Why:

  •  0OI Testing--we have 0OI when we (provably) know something.  In testing terms, this is the "clean" test where we set up, execute, and check a test to prove that the knowledge we put into the code (and by inference the rest of the system associated with the knowledge) is correct.  What we are doing is ensuring the "provably know" component of the definition of 0OI.

  •  1OI Testing--we would never test for this.  Remember that with 1OI, I know that I do not know something.  In code terms an example might be that I know I have a table overflow problem, or a memory leak in my code.  Of course, if I really did know this, I wouldn't run a test to prove it, I would fix it.  And then I would run a 0OI test to provide I'd fixed it.

  •  2OI Testing--this is the rub.  Most of our testing is intended to expose those things that we don't know we don't know.   We run our "dirty" tests to find out if there is anything else the system does that it shouldn't do.  This is also why testing is so difficult; how do we design a test to expose something that we don't know isn't there?  How does a scientist design an experiment to find something she is not looking for?  This is also why testing is an "art"

Two Stages of Testing*                                   

It turns out then that testing always has two stages: find the bug and fix the bug.  These correspond to reducing Orders of Ignorance:

  •  Find a Bug--this is where we convert 2OI into 1OI.  If we have designed a "good" (read: exposes and isolates a defect) test, it flushes out something we didn't know about the system.  I guess we can always assert that there is, or must be, many things we don't know about a system, but unless we can localize our ignorance, all we have is an existential expression of 2OI--it's not useful. 
    The Find-a-bug stage performs the transform 2OI --> 1OI
     

  •  Fix a Bug--one we know there is something we don't know, we stand a chance of determining what that ignorance is and how and where we might find the knowledge that will eliminate it.  Since a "good" test not only uncovers a problem, it also isolates it in terms of its primary characteristics (eg. location, state, input conditions, etc.) that allow us to understand what that ignorance is.
    The Fix-a-Bug stage performs the transform 1OI --> 0OI

In the general case, these two stages also correspond to

  1. Find the question

  2. Find the answer

Not Defect                                                   

Here's a thought:

If the whole purpose of systems development is to learn, to expose ignorance and to increase our knowledge, and...
...the primary source of our problems in software is not what we know, it is what we don't know, and...
...the most problematic things that we don't know are the things we don't know we don't know (2OI), and...
...the primary function of testing is to intentionally and comprehensively expose and repair the things we don't know we don't know, then...

...what we expose is exactly what we want to discover.

These are not "defects" in the general meaning of being defective, they are simply the discovery moments when knowledge is exposed.

The exposure of a bug is "an epiphany of knowledge"--it is the moment when we suddenly realize there is something we don't (didn't) know

 

I deal more with testing in the next article

* There are two types of people in the world: people who assert that everything in the world falls into one of two categories, and everyone else

 

Not Defect