The Standish Report has been a longstanding IT bugaboo, consistently indicating abysmally low project success rates in software engineering (and overshadowing any other research in the popular IT press on this topic). Now Robert Glass, author of Software Runaways, reports in the latest Communications of the ACM on growing evidence in the research community that its methodology is suspect and its conclusions spurious. (This is some of the biggest IT news of the year, I think.) Quote:
"...the Standish Chaos Report could be considered fundamental to most claims of crisis. What do we really know of that study?
"The question is of increasing concern to the field. Several researchers, interested in pursuing the origins of this key data, have contacted Standish and asked for a description of their research process, a summary of their latest findings, and in general a scholarly discussion of the validity of those findings. They raise those issues because most research studies conducted by academic and industry researchers arrive at data largely inconsistent with the Standish findings.
"Let me say that again. Objective research study findings do not, in general, support the Standish conclusions. "
Robert L. Glass, "The Standish Report: Does It Really Describe a Software Crisis?," Communications of the ACM 49:8 (August 2006). (Emphasis added.)
Glass goes on to cite other researchers' evidence that the Standish method was to solicit failure anecdotes from IT executives. It's not surprising they then reported a high incidence of failure! It is not a study in any rigorous sense of the word, if the growing academic consensus is correct.
Thoughts?
-Charlie
I agree that this could be some of the biggest news of the year. I often cite the Chaos report myself as evidence that all is not well with corporate IT. I would not like to think I've been fooled.
Of course, it still remains to be seen whether the criticism of Standish is well-founded, and what the critics' own agendas may be.
In any case, even if the Chaos report has exaggerated the scope of problems, one must acknowledge that IT services as delivered quite often fall short of customer needs and expectations. Many people on the business side at our company express frustration with the results they typically receive from the IT department. That frustration has prompted them to explore alternative methods of running software development projects, and to consider so-called "shadow IT" solutions.
They haven't heard of the Chaos report, so their disenchantment with the status quo cannot be explained by spurious (or suspicious) research methods. As my own experience with IT approaches the 30 year mark, I can say anecdotally that these customers have ample reason to look for better results than corporate IT organizations usually provide. There's more to the problem than can be explained away by questioning Standish's methods.
For that reason, I hope people do not read too much into the questions Mr. Glass has raised. Let's find the answers to those questions, but let's not, in the process, fool ourselves about the general quality of corporate IT services. Standish or no Standish, there is considerable room for improvement.
Posted by: Dave Nicolette | August 10, 2006 at 02:05 PM