Copyright © 2009 Corvus International Inc. All Rights Reserved
The only failure is the failure to learn from failure.
Kevin Everett FitzMaurice
Mental Health Counselor
It's ok to fail;it's not ok to keep failing the same way
Phillip G. Armour
From a workshop on
software process conducted in 1996
Projects are uncertain
One of the primary functions of management is to be "in control". Most managers and most management structures avoid situations where they do not feel that they can guarantee the outcome. The trouble is, this is not possible for software development for the simple reason that, when we start a software project we don't know what we should do to be successful. Well, not in sufficient detail to guarantee success anyway. The reason for this is equally simple: the job of the software project is to figure out what we should do.
Projects can be relatively certain or very uncertain based on the amount, complexity, and type of knowledge we have to get. But no project should be completely 100% certain since that would imply that we have nothing to learn--in which case, we should simply have used what we have built before when we did learn.
Levels of Uncertainty
This graphic shows three levels of uncertainty. The y-axis represents the probability that a project will be completed at some level of resource assignment shown on the x-axis. The x-axis could be budget/effort, project schedule, staffing level, or any other resource allocation.
The graphs are shown as Normal or Gaussian distributions for simplicity's sake, but the probability profile for a software development project is more likely to be a Weibull Distribution (with a shape parameter k between 2 and 3).
Few Variables: The top project has few variables and the variables operate collectively over a fairly narrow range of effect. This means that the project is very likely to be completed at a particular and reasonably predictable value of effort, staff, and time. This project has little chance of significantly over- or under-running.
Many Variables: The bottom project has many variables and/or the variables interact in a way that results in a wide range of possible results. This means the project has a high chance of over- or under-running [Aside: we can see here why a Gaussian is probably not the profile, since projects to not have an equal chance of over- and under-running--there is a much higher likelihood of over-running]
Cumulative Probability: Integrating these curves gives the cumulative probability distributions known in Stats circles as S-Curves.
Most likely = Expected = 50%
If we carefully request expert opinion on the likely resources a project will need, we can view this as the most likely or "50%" estimate. That is, if the opinion understates the task, the project will overrun and if it overstates the task the project project will underrun. If we ask someone to give their opinion and tell them to be neither optimistic nor pessimistic, we should get this number. This result is also has the highest probability
When companies ask their staff to produce an "as-expected" estimate, this is what they would get. If this assessment is true we should expect that across the business of software half our projects would be successful and half would not. This is not the case. In fact, most research shows that only about 20% of projects achieve their goals. So how does the project commitment get from 50% to 20%?
Push Down till You Stop
The figure at right shows the story. The red S-Curve represents the probability of achieving the project goals at some level of time or effort (the x-axis). The y-axis shows the probability at each point along the x-axis. The probability is very low until we hit the "High Risk" point. It rises through the 50% and "Mid Point" and becomes very certain as the graph tracks to the right.
Why is the project "very certain" to the right? Simply because we are allocating more resources of budget, effort, time, or staff and the likelihood of overrunning is less simply because there are more resources.
The 50% point
The "50%" point is the "Expected" level, but it is not the mid-point of the time or effort distribution (this is due to the shape of the Weibull curve, a strict Gaussian would be the same).
When a project manager or estimator pitches the "50%" expected solution, management often subjects the estimate to pressure intended to push the estimate to match the hoped-for goal. To achieve this, management challenges the project's assumptions and encourages the team to consider a more favorable set of assumptions. Common example: "...assume that the project will be staffed by the best people available...", this should shorten the forecast compared to modeling the project with "average" staff. Of course the issue in practice, when the project is actually running, is--does it really get staffed with the best people?
The same optimistic assumptions are mooted for things like turnover, scope creep, tool capability, etc. Note that each of the assumptions may not materially change the way the project will actually run they simply change how we are modeling the project for estimation purposes. We might hope that turnover will be less than "normal" but if it isn't the project will overrun. If we have budgeted for less than normal turnover and we get normal or higher than normal turnover, the project won't have the resources to deal with the additional work incurred by people leaving.
Stop at 20%
This chipping away at the initial "most likely" estimate continues until the combined modeled probability gets down to around 20%. As these assumptions are changed of course the estimate is no longer "most likely" and it gradually becomes "less and less likely". At around the 20% point the project team pushes back, and asserts that the combination of rosy assumptions is simply too unlikely. When they finally push back at management with the same pressure to stop chipping away at the estimate that management is exerting to reduce it, the movement stops. This could be called "Newton's Third Law of Behavior": any behavior will continue until acted upon by a behavior that is equal and in the opposite direction. The 20% probability mark is where the balance point occurs. We know this is true because we know that only one in five projects are successful.
Planning to Fail
This is an interesting business practice. Since the 50% estimate was based on what we expect to happen, and pressure is used to push the projections down to 20% (which removes resources without actually changing the intrinsic risk on the project, of course), it means that...
... if everything works out the way we EXPECT it to, we will fail (!).
Not if something bad happens, not if we run into problems or things we didn't expect, not if we have bad luck--but simply if it works out as we expect--this project will not be successful.
It is a very strange way to run a business. We are literally planning to fail.