Cases from The Heart of Innovation

Note: This is Part 5 in a short series of essays on Understanding Customer Demand. Read Part 4 here.

In the previous instalment we examined the ideas of ‘Deliberate Innovation’: a demand framework that I argued is the first major contribution to our understanding of the topic in two decades. The ideas were first published in the 2023 book The Heart of Innovation (HOI); last week’s essay was intended to give you a comprehensive summary.

In this instalment, I want to introduce you to a few cases from HOI that I think are really illustrative of those ideas. If you’re new to Commoncog, you might want to check the notes on Commoncog’s Calibration Case Method. But the basic idea is simple: the more examples you look at, the better your calibration of how these concepts work in the wild. In business, it’s even more important to pay attention to cases (as instantiations of these concepts) over frameworks, because every situation you will encounter is going to be somewhat unique. As a result, you need a library of concept instantiations to compare against in your head; learning the framework alone is never enough.

We’re going to look at three cases today. I expect you to click through and read each case before coming back to this essay.

Let’s get started.

The Discovery of RAID

The first case concerns the discovery of RAID — otherwise known as ‘Redundant Array of Independent (or Inexpensive) Disks’. RAID turned out to be a huge ‘not not’ for the hard disk drive industry. But this discovery was a total accident.

Around the same time that IBM was commercialising GMR, David Patterson and his colleagues at UC Berkeley had come up with a different technology. While GMR targeted areal density — which led to increases in storage size, Patterson was looking at speed. The Berkeley team had come up with a way of linking multiple drives together, based on a system they called RAID (Redundant Arrays of Inexpensive Disks). The computer would see the RAID system as one hard disk drive, even though the system was made up of multiple disks. Patterson thought that by distributing the data onto two (or more) disks that were spinning, you could cut down on the time till the next bit was read, speeding things up. 

But when Patterson demonstrated his RAID technology, something unexpected happened. The purpose of the demo was to demonstrate the increased speed of RAID systems — as was the conventional wisdom at the time. As an afterthought, at the end of the demo Patterson dramatically pulled one of the hard disks out of the system. Because the data was duplicated on each disk, the system kept working, just a little slower.

It was this afterthought that caught the attention of the market.

In an interview years later, Patterson said: 

“We were still performance-oriented, thinking RAID was for performance, so we were shocked to see somebody write this up in Byte magazine. The PC community was obviously not so performance-oriented as it was dependability oriented, and they thought, Hey, less-expensive dependable computing. 

It really just took off after that. EMC decided to build mainframe storage out of PC disks. Compaq had RAID early, and Data General. And of course [eventually] IBM had its own.”

EMC took the lead on RAID systems, largely on the basis of dependability. (To be fair, EMC was already the market leader in enterprise data solutions by this point, having taken the lead in the early 90s). IBM was late to the party. Worse — for IBM — RAID systems were lucrative. The demand for dependability was such that RAID boxes in those early years came with eye-watering price tags. In the same interview, Patterson continued:

“One of the surprises about RAID was it was so expensive. The I in the name when we coined the term was for inexpensive disks. But the system was so expensive, that was kind of awkward for marketing people. So Randy [Katz, of Berkeley] blessed the change to independent for I. Since the RAID boxes weren’t cheap, that was probably a better name.

Notice how the gap in this scenario was invisible before Patterson’s fateful demo. The hard disk drive industry was 40 years old at that point; you would have thought that the two metrics that everyone watched — speed and size — would have been the only two measures that mattered.

And yet Patterson proved otherwise. He fell backwards into authentic demand, it seems like.

The other thing that’s notable is that EMC exploited this way before IBM did. Technological advantages don’t last, of course — at most a technical edge gives you an advantage that you may exploit for a few short years. But EMC exploited RAID for all it was worth: achieving more than 50% market share for data storage solutions in the IBM mainframe market by the end of the 90s. It’s not too much to say that EMC’s current leadership position in the enterprise data storage space was due to a breakthrough RAID product — the Symmetrix 4200 system — which it sold as aggressively as it could in the decade that it was invented.

With the benefit of hindsight, exploiting RAID was obvious. But clearly it wasn’t at the time. The authors of The Heart of Innovation write:

To Patterson’s surprise, the performance issue didn’t drive the market, and to IBM’s surprise, miniaturization wasn’t a key advantage either. Instead, reliable, dependable, easy-to-manage RAID systems took off, eventually evolving toward server farms and creating the infrastructure behind social platforms, streaming videos, and the modern web.

(…) Authentic demand for reliability was hiding in plain sight. Everyone in the market knew it was important — of course hard disks had to work. The critical metric was “mean time between failures” (MTBF). Acceptable MTBF standards were well known and available, and all the new disk drives from IBM, EMC, and other competitors met them. In hindsight, though, it’s clear that if the MTBF is steady while the disk capacity goes up, there’s trouble ahead. If a drive holding sixteen gigs of important company information fails, the company will have a bigger problem than if a one-gig disk were to fail. For the overall risk to remain the same, the MTBF standards would have to increase in step with disk capacity. No one in the market was paying attention to that — until the fateful moment when David Patterson yanked a live hard disk out of a RAID system.

It wasn’t that the people at IBM just happened to be looking the other way at that moment and failed to recognize the authentic demand; that mistake was a long time coming. Patterson could casually pull a hard disk out of a RAID system thanks to his pioneering work on software for managing hard disks. Ironically, there was a team at IBM that had done a lot of similar work, and even owned some important patents in the area. But that was software. IBM was a hardware company where software only played a supporting role. The groups at IBM responsible for moving research to development to market were hardware groups. They didn’t recognize what they had.

And we can imagine, now, why this was a not not. If you were an enterprise buyer, you would have looked at your fleet of data storage machines, each drive a spinning chunk of metal rotating thousands of times a second, and think to yourself: god, if any one of them failed, what sorts of enterprise data might I lose?