SE Radio 250: Jürgen Laartz and Alexander Budzier on Why Large IT Projects Fail

Alex Budzier of the Oxford Saïd Business School and Jürgen Laartz of McKinsey Berlin join Robert Blumen to discuss their research on large IT project failures. The show covers: What is a “large” project? What is the definition of failure? Cognitive biases and project failures. Are some attributes of projects predictive of failure? The catastrophic “black swan” failure mode.  How do projects spin so badly out of control? What are the warning signs that a project is going into a black swan mode? Why are massively late and over-budget projects not cancelled? Are mission-critical “bet the business” projects more likely to end badly? Can costs be accurately estimated in advance?  The role of cynical political calculations in cost overruns.  Can projects be estimated bottom up or are historical methods more accurate?  Are unclear requirements responsible for failures? Do agile practices work at scale?  And finally, how can companies succeed in critical projects?

Show Notes

Related Links

• Alexander Budzier’s home page at the Saïd School https://www.sbs.ox.ac.uk/community/people/alexander-budzier
• Research on IT projects and Risk at the Said School
• McKinsey Quarterly http://www.mckinsey.com/insights/mckinsey_quarterly
• McKinsey Business Technology portal http://www.mckinsey.com/insights/business_technology
• Delivering large-scale IT projects on time, on budget, and on value http://www.mckinsey.com/insights/business_technology/delivering_large-scale_it_projects_on_time_on_budget_and_on_value
• Why IT projects really fail http://www.cio.com.au/article/533532/why_it_projects_really_fail/
• Why Good Projects Fail Anyway https://hbr.org/2003/09/why-good-projects-fail-anyway
• Why do IT projects fail? http://economia.icaew.com/business/january-2015/it-failures
• Why software fails http://spectrum.ieee.org/computing/software/why-software-fails
• A Reflection on Why Large Public Projects Fail by Kjetil Holgeid and Mark Thompson https://www.jbs.cam.ac.uk/fileadmin/user_upload/programmes/emba/downloads/A_Reflection_on_Why_Large_Public_IT_Projects_Fail_-_Kjetil__Mark_Thompson_s_chapter.pdf
• Alexander Budzier at Google Scholar https://scholar.google.com/citations?user=DzngGSwAAAAJ&hl=en

Transcript

Transcript brought to you by innoQ

At some point in their career, most software engineers have worked on a failed or cancelled project. But why do some IT projects fail? Is the perception of pervasive IT failure accurate? The guests of Software Engineering Radio Episode 250, Jürgen Laartz and Alexander Budzier, teamed up to tackle these questions in their research. Budzier is a faculty member in Oxford University’s Saïd Business School, and Laartz is a director of the McKenzie and Company Business Technology Office in Berlin, where he leads the company’s global IT architecture practice.
Topics not covered here because of space include how agile practices affect large IT projects, how to control the cost of failure, and cognitive bias’s role in project failure. Please contact us with your ideas and thoughts about this column, or any aspect of the show, at [email protected]. —Robert Blumen

Robert Blumen (RB): How large is a “large” IT project?

Alex Budzier (AB): Size is something, particularly in IT projects, that is really difficult to judge. If you’re building a very large infrastructure, like a train station, a road, or a bridge, then we typically classify a large project as one costing more than a billion dollars. But it’s difficult to say “It needs to be above a specific threshold” to be large. What we should be talking about is, “What are the problems in IT projects?” or “What is driving some of the difficulties of managing IT projects?” There are two factors besides size here: time span and whether the project is transformative to the organization. “Large” projects are either very long projects or transformative ones.

Jürgen Laartz (JL): One of the counterintuitive results of our joint project was that we actually didn’t find any discriminating sector, whether a project is large in the classical sense or not. One of the key questions we asked was, “Does size actually matter in terms of success rates or failure probability for IT projects?” We found that this is not the case.

AB: It’s interesting because size is the dominant criteria to define a governance structure. Most organizations we’ve worked with say that if you’re above a certain budget size, then you’re reporting to this specific board. If you’re above that budget size, then your project is seen as a higher risk, so you must report directly to the chief information officer or the [executive committee]. Often the governance structure, which is based on the need to manage high-risk projects, is by size (budget size in particular). Yet, we saw that this doesn’t have a lot to do with the success or risks of these projects.

RB: How do we classify a project as a failure?

JL: Failure is a difficult attribute to use. It’s highly subjective. A real failure is when a project is terminated—that’s undisputable. But even if a project is terminated, in some cases it’s due to a change in requirements. The question is what happens with projects. A real failure for an IT project is when it doesn’t create the impact it should have delivered. It doesn’t support a business process, or the underlying software doesn’t deliver on the analytics or whatever the task of the specific software was.
We’re often talking about cost and time overruns or business case nondeliverables. These are not necessarily linked to failures of projects; they’re linked to planning accuracies and to delivery capabilities in organizations. The question, then, is whether the failure in itself is the right category to look at.

AB: You could easily ask the question, “What constitutes success?” There’s a great example (although not an IT project) of the Norwegian Army building defense stations along the coast of Norway to defend against the invading Soviet Navy. They built these stations in the late 1990s. The project was delivered on time and on cost. It was delivered exactly with the scope prescribed, and it was delivered in the quality they wanted. But it only went into operation for about six months because the threat wasn’t there anymore; the world had moved on. The project delivered the scope and the quality on cost and on schedule, but you wouldn’t say the project was a success. It was a massive waste of taxpayer money. According to those narrow definitions of success and failure, that one is a true failure that would have been counted as a success.

RB: Is it valid to classify projects into IT failures versus business failures? In my view, if the project was delivered but is no longer needed, that is a business failure. Is that a valid distinction?

AB: The real line in the sand is when the project gets terminated and nothing ever goes live. That happens, according to some surveys, in 8 to 10 percent of all projects. We also know that the key reason why projects get aborted is because they lost the support of the sponsorship; in 80 percent of the cases, that’s the number-one factor. Technology struggles only happen in about 30 percent of cases.
IT projects are there to deliver an outcome, and that’s how they should be judged. If the project is late and the outcome was needed earlier, it’s not ideal. If you needed a lot more money to get to the outcome, that’s also not ideal. But I would not always classify it as a failure.

RB: With all the uncertainties about the future and changes that can occur in business, we can expect that some projects will fail. Do you have a point of view about whether too many IT projects fail? Are many of these failures avoidable?

JL: As many as 80 percent of projects are terminated because the project lost the support of the stakeholders, mainly because of changes in the business environment or new organizational or process structures.

AB: We studied about 4,300 projects. We can’t always say where there was a success or a failure. However, we can say whether the projects were able to produce accurate estimates and were accurately planned. A whole lot of money and resources is spent governing and supervising these projects. So, the question is whether that money was wasted.
We see from our data that five out of 10 projects deliver on budget and on time. But we also see that on average, cost overruns double their cost and about 34 percent of projects are delivered late. That really begs the question “What is the problem in IT?” because every second IT project is just doing fine.
It’s not so much that there is a high rate of failure and a low rate of success, but that failed projects drive up the average, despite a lot of projects delivering on time. And that’s the true risk in IT. In other words, we really need to bring down the variability between how different IT projects are faring.

RB: There’s a particular catastrophic failure mode that exemplifies failed projects bringing up the average, which you refer to as a “black swan.” Can you explain what this is?

AB: This is from a book called The Black Swan by Nassim Nicholas Taleb, in which he discovers some cases where disruptive events have propelled humankind forward. He borrows the metaphor from an old text by John Stuart Mill, where he tells the story that in the 1700s, London had a very famous saying that “all swans were white.” Some Dutch explorers came back from Australia with a pair of black swans, and suddenly a fact that had been verified a gazillion times over the history of Europe was falsified by the existence of two birds.
It’s a very small event with very big ripple effects. And nobody saw it coming. Afterward, everybody was thinking, “Why the heck did we ever think that swans can only be white? Why can’t they be all colors?” Taleb compares that to the traders in financial markets who believe they’re in a very certain world where they can quantify the risks, but in reality they are in a world of randomness. Sometimes massive losses occur, which he calls black swan events.
Interestingly, when we looked at our projects, we found a very similar statistical signature in the cost and schedule performance of IT projects that Taleb discovered in financial markets. That’s the interesting analogy: in hindsight, some of these small or seemingly small but high-impact projects in a big portfolio, the ones that cause a lot of disruption, seem to make perfect sense. Most organizations didn’t see them coming.

RB: Can you recall any black swan examples from your work?

JL: I recall a lot of examples, but I can’t share them because of the confidentiality of the consulting work. A lot of very high-profile examples are out in the open and are published. You can look at K-Mart, for example, or healthcare.gov. I think the visible ones are the tip of the iceberg. We’ve done portfolio reviews together for companies, and there is not one company that doesn’t have at least one or two black swans in their portfolio. In foresight, the organizations weren’t able to determine the risk candidates; in hindsight, everything is clearly there, which has a lot to do with biases in organizations.

RB: I read about one project that was budgeted for $5 million that in the end cost $200 million and set in motion the bankruptcy of the entire firm. How can a project get from 5 to 200 without somebody along the way saying “stop”?

JL: A lot of what we are talking about here has to do with psychology. You will find optimism-biased project plans that permeate into the project’s execution. Not stopping projects has to do with steering bodies that have an optimism bias. They see some progress, maybe a light at the end of the tunnel, but the tunnel gets longer and longer. While they do not lose sight of the light, they don’t come through the exit either. Some organizations don’t have the capability to define a cutoff point for these kinds of events because they don’t see the totality; they see only the next incremental step.

AB: An alternative and very cynical explanation is that the initial estimates were never realistic. Everybody knew right from the start that the project couldn’t be delivered at that price point, but maybe there was no more funding available, so everybody pretended that this would be feasible. As soon as you get going and show first incremental steps, it’s really hard for an organization to pull out again. We’ve seen very few projects getting upended, even if they escalate massively. The tendency and psychology is to stick with them. This is not only a result of optimism but also very sinister politics and organizations that are pledged to secure funding and start the project because chances are it won’t be stopped.

RB: That calls into question whether some of these projects are even over budget, if everyone knew that the budget was deliberately understated.

JL: You see this not only in IT projects but in a lot of other types of projects as well. The thinking is that this needs to fit into budget at the start and then we’ll see what happens. We’ll free up more and more money when we’re on the way. From a tactical point of view, this might sound okay. But this has a ripple effect, and the planning is not done in the right way, because it doesn’t stop at the technical. The entire plan needs to represent the lower budget that is assigned to a project. If you get started on the wrong foot, even if everybody knows that the numbers aren’t right, and if you don’t adjust the planning to what you believe is right, you will always run into trouble.

AB: We found that there are very simple steps organizations, project managers, and decision makers can take to prevent this. The best analogy is if you were to remodel your kitchen, you could go to a studio and have someone design your kitchen. They’ll quote you a price that you will find acceptable, and later on there will be a lot of changes and upgrades that will be added onto the price point. The alternative approach is to go around your neighborhood and chat with your neighbors who have recently remodeled their kitchen and ask them what they actually paid in the end.
What we see with IT projects is that the second approach often gives you a lot more accurate insight into the true costs of a project. You can take the political bias and the optimism bias out of those estimates by simply asking the question, “Of the last 20 people who wanted to do exactly this or similar projects, what range were they at?” If everybody pays at least $200 million to implement a certain type of IT project, how on earth could you do that for $5 million? You get a lot more accuracy by taking an independent, data-driven view of the project.