In 1995, a young Harvard Business School Professor co-authored an article in Harvard Business Review, “Disruptive Technology: Catching the Wave.” He and his co-author proposed a new causal mechanism that explained the surprising failure of highly-regarded companies. The most punishing innovations, they argued, were the ones that were easy to dismiss at first blush — simple, affordable solutions that took root outside the mainstream market. The authors called these “disruptive” solutions and provided a straightforward prescription for leaders looking to turn disruption into an opportunity. They suggested that companies should find a customer who loved the disruptive solution despite its limitations and create a separate organization to commercialize it.

Of course, that young HBS professor was Innosight co-founder Clayton Christensen. Since then, he has written over a half-dozen books and many more Harvard Business Review articles, almost all of which touch on disruption in some way. Academic journals have dissected the disruptive innovation theory and hundreds of thousands of students around the world have seen Christensen’s famous model.

Yet, the innovator’s dilemma persists. Just ask executives at Blockbuster Video, Sony, Nokia, Microsoft, Hertz, Kodak, Delta, and nearly all newspaper companies. That’s not to say that there haven’t been success stories. But they’re notable because they are exceptions.

So, why has this dilemma persisted?

Capital markets is one explanation. As this argument holds, the short-term pressure of the capital markets, coupled with management incentives tied tightly to stock prices, make it hard for companies to investment in new growth businesses. Even if companies know what they need to do, their investors won’t let them. Investors aren’t necessarily irrational, since they could presumably back up-and-coming disruptors themselves. There is at least one strike against this argument. In markets such as India, Korea, Japan, and China, companies that have different corporate governance models seem equally likely to suffer from the innovator’s dilemma.

Perhaps the root problem is leadership limitations. Leaders in established companies seeking to drive disruptive growth have to meet the challenge laid down in 1935 by F. Scott Fitzgerald: “The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time and still retain the ability to function.” Yet, in Chapter 9 of The Silver Lining, I described a stream of research suggesting that only about 5% of leaders have the ability to pass Fitzgerald’s test. Every once and a while a Steve Jobs, a Jeff Bezos, or an A.G. Lafley appears, but perhaps their rarity leads to the dilemma’s persistence.

Maybe the real challenge lies within. Over the past few decades there has been a fascinating set of research into cognitive biases that lead smart people to make bad decisions.* These biases are particularly acute for companies trying to drive disruptive innovation. Consider the “halo effect,” which holds that people who are demonstrably good at one thing are perceived to be good at non-related tasks. The halo effect leads companies to assuming their best operators can seamlessly shift into innovation work. Some can, but many cannot. Confirmation bias, disaster neglect, the fundamental attribution error and many others make it easy to simultaneously discount the need to respond to disruptive threats and overestimate the organization’s ability to step into new markets.

There are surely other explanations — for instance, people don’t always have high levels of interest in transforming a system that confers certain powers on them — but my own view is that the dangerous combination of leadership limitations and cognitive biases makes the innovator’s dilemma an intensely difficult problem to solve.

Perhaps further clarity in the root cause of the persistence of the innovator’s dilemma can help increase the number of success stories. Any other ideas?

* Dan Ariely, Michael Mauboussin, Nobel Laureate Daniel Kahneman, and Duncan Watts all write accessibly on the topic.