Copyright © 2009 Corvus International Inc. All Rights Reserved
Measure what is measureable,
and make measurable what is not so.
Galileo Galilei (1564 - 1642)
Phillip G. Armour
Communications of the ACM - Privacy and security in highly dynamic systems, 2006
Three Million People
According to the US Bureau of Labor Statistics, over three million people are employed in in the business of software (as of May 2005). Given that, there is an astonishing lack of solid data on how good we are at building the stuff. Luckily there are people who make it their business to do this.
And even more luckily, they are publishing their results.
Quantiative Software Management
QSM was founded in 1978 by Larry Putnam Sr. Larry is, along with such luminaries as Barry Boehm and Capers Jones, one of the pioneers of software project estimation. QSM develops and markets the SLIM-Estimatetm estimation tool suite. SLIM is, in my opinion having used all the estimation tools known to the human race, the best in class. But I digress.
Along with providing this tool, QSM also collects data from its customer base.
In 2006, they published The QSM Software Almanac: 2006 IT Metrics Edition which contains some really interesting data.
The QSM Software Almanac
This study looked at a sample of 564 recent IT projects from 31 corporations in 16 industries in 16 countries. The mean size of projects was slightly under seven people though the median size was between one and three people. The average duration was around eight months requiring 58 staff months of effort and delivering a median code size of 9.200 LOCE. Here are the findings:
Smaller is (much) better
Large teams (29 people) create around 6x the defects that small teams (three people) do. Yet the large team delivers the same amount of output in only 12 fewer days (less than 2% faster) !
Comparing the best-in-class to the worst in class QSM measured:
Effort: 15x less effort
Schedule: 1/5th of the duration
Team Size: much smaller teams
Teams and projects that perform poorly also don't tend to collect careful data on how poorly they perform. Therefore, QSM determined that it is not possible to empirically measure the quality different between good and bad teams
Decreasing Code Size
Over the last 20 years, delivered code size per project has decreased substantially, from 80KLOC in the mid 1980s to 55KLOC in the the mid 1990s to around 28KLOC in 2005.
Bigger is better
In contrast to the the effectiveness of smaller teams, larger system sizes are associated with higher levels of productivity. This is likely due to the increased resource allocation given to large important systems.
Productivity is going down
One of the most interesting observations is that "productivity" (as measured by SLIM's "Productivity Index" or PI) has been going down since 1997 and is now at the same levels that were measured in the 1980s. This is likely due at least in part to the more complex systems we build today and the decreased code size (PI is usually measured or normalized against the delivered code size) driven by both package implementation and the use of higher level languages.