Product Development as Iterated Taste

Working Backwards is the first book to describe how Amazon really works, written by two insiders who were in the room when the techniques were first invented. I finished the book a few weeks ago, and I’m still chewing on many of the ideas.

I want to focus on one particular idea this week. In Chapter 5, the authors — Colin Bryar and Bill Carr — describe something they call the ‘Working Backwards process’, which is Amazon’s method for betting on new products. (Yes, this is both the name of the process and the name of the book; it’s a little confusing).

Bryar and Carr describe the process like this:

(…) what turned out to work best was relying on the core Amazon principle of customer obsession and a simple yet flexible way of writing narrative documents. These two elements form the Working Backwards process — starting from the customer experience and working backwards from that by writing a press release that literally announces the product as if it were ready to launch and an FAQ anticipating the tough questions. While this next section describes the evolution of Working Backwards as seen through the experience of the digital team, a handful of other teams went through a similar process. Bringing together the experience of these teams enabled us to hone and refine Working Backwards into its final form.

One way you can think about the Working Backwards process is that it is a response to the fundamental difficulty of launching a new product. This problem is actually pretty universal: say you want to launch a new thing, but creating a new thing costs hundreds of man-hours and potentially millions of dollars. With stakes that high, you don’t really want to fail. So what do you do?

For many years the answer was to ‘write a business plan’ or ‘do market analysis’ or ‘conduct demographic tests’. But in the early 2010s, a new orthodoxy emerged with Eric Ries’s The Lean Startup. “Go build a ‘minimum viable product’,” Ries argued, writing in the first person:

At this point in our careers, my cofounders and I are determined to make new mistakes. We do everything wrong: instead of spending years perfecting our technology, we build a minimum viable product, an early product that is terrible, full of bugs and crash-your-computer-yes-really stability problems. Then we ship it to customers way before it’s ready. And we charge money for it. After securing initial customers, we change the product constantly — much too fast by traditional standards — shipping new versions of our product dozens of times every single day.

(…) Traditional business thinking says that this approach shouldn’t work, but it does, and you don’t have to take my word for it. As you’ll see throughout this book, the approach we pioneered at IMVU has become the basis for a new movement of entrepreneurs around the world. It builds on many previous management and product development ideas, including lean manufacturing, design thinking, customer development, and agile development. It represents a new approach to creating continuous innovation. It’s called the Lean Startup.

Ries’s book was published in 2011. This means that it’s been 10 years since The Lean Startup first hit shelves, and 10 years since it changed the world of product development. So much so, in fact, that the Lean Startup model has pretty much become the orthodoxy for launching a new product or company today. And the idea itself is simple to articulate: build an extremely rough version one, launch it, find some users, and then iterate on the product as quickly as you can. You are then expected to use the information generated by your iterations to either stay the course and keep tuning the product, or — in rarer circumstances — to change your approach altogether; Ries calls this ‘pivoting’, because you’ll have to rethink your ‘strategy’ (which in Lean Startup parlance consists of your product roadmap, your competitors and partners, your business model, and your target customer).

Talk to a startup person today and you'll likely hear things like ‘MVP’ or ‘pivot’ or ‘product/market fit’; these ideas have spread so far that they’ve become conventional wisdom. But this, in turn, implies something interesting: whenever you come across a successful company doing things differently from the standard ‘Lean Startup’ method, it is always worthwhile to pause, and pay attention, and ask: “wait, why does this work?”

Amazon’s ‘working backwards process’ is a pretty fascinating example of an alternative that works — I should know; I’ve been collecting alternatives to the Lean Startup for a number of years now.

(In all honesty I’ve only ever used the Lean Startup methodology — enough times, in fact, to know that it has shortcomings. I’ve not tried any of the others).

The method is especially interesting because of its impact on the business — Amazon is, after all, in more industries and product categories than pretty much any other Western tech company. It's a bit ridiculous that it is able to enter, invent and dominate so many niches repeatedly; on this basis, I think the Working Backwards process belongs on the pantheon of other successful product development methods, alongside The Lean Startup, Apple’s Creative Selection Process, and Pixar’s Braintrust.

And I think there's a fair amount we can learn from it.

Amazon’s PR/FAQ Method

Amazon’s Working Backwards method is simple to describe, but difficult to do. You start out by writing a press release, which follows a very particular structure. You move on to write an attached FAQ document, which addresses a bunch of internal and external issues. These issues include (but aren’t limited to) things like total addressable market, per-unit economics, bill of materials, P&L, key dependencies, and technical feasibility. The total document — both PR and FAQ — should not exceed six pages in length.

Creating this document is quite the exercise — you write it with the understanding that every single sentence, assertion, or assumption that you include would be uncovered and interrogated by the people reading it. Whenever possible, you should get the facts; the authors explain: “a good PR/FAQ is one in which the author has clearly considered and grappled with each of these issues, seeking truth and clarity on each.”

Once submitted, your PR/FAQ is reviewed in a one-hour meeting with the relevant stakeholders present. The first 20 minutes is dedicated to reading the document in silence. (Amazon assumes that people take three minutes to read each page; this explains the six-page limit). When everyone has finished, the writer asks for general feedback. The most senior attendees tend to speak the last, to avoid influencing the others.

This process sounds really simple, but think about what it must feel like to write one. The authors describe a process early on in the PR/FAQ cycle for S3:

“(…) S3 would be a tiered monthly subscription service based on average storage use, with a possible free tier for small amounts of data. Customers would choose a monthly subscription rate based on how much data they typically needed to store—Simple Storage Service with simple pricing. We hadn’t worked out the exact details of the tiers and the prices for each tier, but you don’t have to do that in early iterations of the Working Backwards process. The engineering team was ready to move on to the next question.

Except that day we never got to the next question. (emphasis added). We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates? How often would updates happen versus reads? How many customers would need simple storage (can easily be re-created, stored in only one location, not a big deal if you lose it) and how many would need complex storage (bank records, stored in multiple locations, a very big deal if you lose it)? All those factors were unknown yet could meaningfully impact our costs. Since we didn’t know how developers would use S3, was there a way to structure our pricing so that no matter how it was used, we could ensure that it would be affordable to our customers and to Amazon?

So imagine that you’re a software engineer, and you’ve just been told that you’re going to build an exciting new product for an exciting new business at Amazon. You write a PR/FAQ with your team, checking through all of your assumptions, only to get sideswiped by the execs in your first meeting. One of them points to the line in your doc that says ‘S3 will be a tiered subscription model’ and goes  “now, hold on a second …”, and then you spend 40 minutes just arguing about pricing.

Eventually — Bryar doesn’t mention how long — the stakeholders decide on something called ‘cost-following pricing’:

(…) “the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. “Cost following” means that your pricing model is driven primarily by your costs, which are then passed on to your customer. This is what construction companies use, because building your customer’s gazebo out of redwood will cost you a lot more than building it out of pine. If we were to use a cost-following strategy, we’d be sacrificing the simplicity of subscription pricing, but both our customers and Amazon would benefit. With cost following, whatever the developer did with S3, they would use it in a way that would meet their requirements, and they would strive to minimise their cost and, therefore, our cost too. There would be no gaming of the system, and we wouldn’t have to estimate how the mythical average customer would use S3 to set our prices.

So, ok! Good! You’ve worked out pricing! Now you can turn your attention to more interesting technical questions, like S3’s potential failure modes —

Would the most important cost drivers for S3 be the cost of storing data on the disk? The bandwidth costs of moving the data? The number of transactions? Electrical power?

But of course you cannot, you need to go think about the right costs to follow. So off you go to do the research — meaning you test, instrument, and measure various parts of Amazon’s existing infrastructure so that you may present an updated pricing model for the next PR/FAQ review meeting. You eventually figure out that storage and bandwidth are the most expensive cost factors; in the end, everyone decides on storage and bandwidth as the right costs to follow.

This turns out to be mostly right. After S3 launched, Amazon discovered that they had missed out on one other cost factor. As CTO Werner Vogels explains:

An example in the early days where we did not know the resources required to serve certain usage patterns was with S3: We had assumed that the storage and bandwidth were the resources we should charge for; after running for a while, we realised that the number of requests was an equally important resource. If customers have many tiny files, then storage and bandwidth don’t amount to much even if they are making millions of requests. We had to adjust our model to account for the all the dimensions of resource usage so that AWS could be a sustainable business.

Nevertheless, the authors argue that using a cost-following strategy meant they could correct their mistake and adjust pricing relatively easily. If it hadn’t been for the PR/FAQ process, they would have had to make a far bigger post-launch change from their original idea of ‘simple subscription pricing’.

And so the end result of a PR/FAQ meeting is often a decision to continue working on the … PR/FAQ. This is not a typo: most PR/FAQs are rejected, and the ones that pass often generate further questions that need to be answered — like with the pricing example above. In the end, it took 18 months of PR/FAQ iterations before Amazon’s software engineers wrote the first line of code for AWS. (I relayed this anecdote to a programmer friend of mine, and we both agreed that we would much rather quit than spend 18 months iterating on a Word document).

Bryar himself writes:

The software engineers in the PR/FAQ meeting where we discussed pricing were getting antsy. One of them pulled me aside afterward and said, “We’re software engineers, not pricing specialists with MBAs. We want to write software, not more Word documents!”

So why does this process work? Bryar and Carr explain:

The fact that most PR/FAQs don’t get approved is a feature, not a bug. Spending time up front to think through all the details of a product, and to determine—without committing precious software development resources—which products not to build, preserves your company’s resources to build products that will yield the highest impact for customers and your business. (emphasis added)

Another one of the biggest benefits of a written PR/FAQ is that it enables the team to truly understand the specific constraints and problems that would prevent a new product idea from being viable and aligning on them. At that point, the product or leadership team must decide if they will keep working on the product, addressing the problems and constraints surfaced by the PR/FAQ and developing solutions that will potentially make the product viable, or if they will set it aside.

(…) If, after the PR/FAQ process, the leadership team still believes in the product and wants it to become a reality, the process will have given them a thorough understanding of the problems that would need to be solved in order to move forward with it. Perhaps a problem can be solved through an acquisition or a partnership. Perhaps it can be solved with the passage of time—new technologies may become available, or the costs of the technology might come down. Perhaps the company decides that the problem or constraint is solvable, that the solution will require risk and cost, and that they are willing to assume that risk and cost because the TAM is large and therefore the potential rewards are great.

Does this process work 100% of the time? No, it doesn’t. The authors argue that the PR/FAQ process merely increases the odds that your product would succeed; it doesn’t guarantee it. They explain that the Fire phone, for instance, was a flaming wreckage of a product, and it went through the PR/FAQ process all the same. So it isn’t magic.

But there’s an interesting side-benefit to the entire process that I think is worth chewing on: Bryar and Carr say that by forcing multiple iterations on a PR/FAQ, you implicitly gain the approval of everyone who reviews it, demands iterations on it, and subsequently approves it. This then means that when a product passes the PR/FAQ process but subsequently fails in the market, people are more likely to take collective responsibility.

When you see it like this, the PR/FAQ begins to look like a nice political tool — but the good kind, the kind that aligns incentives between execs and inventors.

Wrapping Up

I am under no delusions that the PR/FAQ process is universally applicable — in fact, I think it is highly dependent on cultural values that exist in Amazon. I say this partly because the authors spend the entire first chapter of Working Backwards arguing this point, but I also say this because it is common sense: org processes can rarely ever be copied wholesale, and may only be used with some adaptation to your specific context.

If you take a step back and squint, however, you can see certain commonalities between the Working Backwards process and others like it.

For instance, Apple’s Creative Selection process is driven by a demo-first culture — you are invited to sit in demos only if you have demonstrated good taste in product development in the past. This iterated demo process is how Apple is able to innovate on both software interfaces and hardware designs, and how they are able to release ridiculously polished products on day one.

As author and former Project Purple developer Ken Kocienda describes it, in one of his own Steve Jobs demos:

The people sitting with Henri (Lamiraux) on the couch were the true software development inner circle. They were the few Steve wanted around him to advise, consult, and bounce ideas around as he reviewed demos. They had done this for the iPhone, and now they were there for the iPad too. Each person had earned their place, and kept it, by consistently providing feedback that helped make the products better.

Sitting to Henri’s left on the couch was his boss, Scott Forstall, then the senior vice president of iOS software engineering. Scott reported directly to Steve, and he was the one giving me this chance to demo in Diplomacy (note: this was the the name of the meeting room Kocienda was currently in). Scott expected me to keep it concise and on point when it was my turn to go. He didn’t tell me to do that—proper conduct was implicitly communicated through the success and failure of the earlier-stage demo sessions Scott ran himself, where he was the top executive in the room. The stakes in Diplomacy were obviously higher, now that his boss was in the room, and since Scott was my sponsor, my demo would reflect on him. Given what I took to be my new probationary status in the Diplomacy inner circle, I imagined that one bad demo might cause Scott to rescind my membership. Even though he never spelled it out, for all I knew, I might be a single boneheaded comment away from never being invited back.

Or take Pixar’s Braintrust structure, which workshops story ideas as ‘reels’, showing them to a core group of storytellers for debugging before committing resources to production. Ed Catmull writes, in Creativity, Inc:

After the release of Toy Story 2, our production slate expanded rapidly. Suddenly, we had several projects going at once, which meant that we couldn’t have the same five people working exclusively on every film. We were not a little startup anymore. Pete was off working on Monsters, Inc., Andrew had started Finding Nemo, and Brad Bird had joined us to begin work on The Incredibles. The Braintrust had to evolve, then, from a tight, well-defined group that worked on one film together until it was done to a larger, more fluid group that assembled, as needed, to solve problems on all our films. While we still called it the Braintrust, there was no hard-and-fast membership list. Over the years, its ranks have grown to include a variety of people—directors, writers, and heads of story—whose only requirement is that they display a knack for storytelling.

(…) As I’ve discussed, first we draw storyboards of the script and then edit them together with temporary voices and music to make a crude mock-up of the film, known as reels. Then the Braintrust watches this version of the movie and discusses what’s not ringing true, what could be better, what’s not working at all. Notably, they do not prescribe how to fix the problems they diagnose. They test weak points, they make suggestions, but it is up to the director to settle on a path forward. A new version of the movie is generated every three to six months, and the process repeats itself. (It takes about twelve thousand storyboard drawings to make one 90-minute reel, and because of the iterative nature of the process I’m describing, story teams commonly create ten times that number by the time their work is done.) In general, the movie steadily improves with each iteration, although sometimes a director becomes stuck, unable to address the feedback he or she is being given. Luckily, another Braintrust meeting is usually around the corner.

I think one way you can think about this is that for a product to succeed, lots and lots of things have to go right. And if even one critical thing goes wrong — be it software design, or some hardware flaw, or market size, or manufacturing costs — the odds are good that the entire product flops. And so when you see enough examples of new product development, you’d quickly realise that all of the product development processes that I’ve described — Lean Startup, Working Backwards, Creative Selection, and Braintrust — are simply ways of iterating cheaply through an idea space, with sufficient feedback, in the hopes of checking enough boxes for success.

Hopefully, you do this at a very low cost. Putting together a UI demo is a hell of a lot cheaper than putting together a full phone; throwing together a ‘reel’ is cheaper than rendering a full movie, and writing iterated PR/FAQs is a heck of a lot easier than retooling an AWS product post-launch.

If there’s anything we’ve learnt from the decade after Lean Startup, it is that sometimes, even building an MVP can be too expensive. (And sometimes having a core group of differentiated evaluators can be just as good, if not better, than getting feedback from users). I find it interesting that so many different companies have independently created similar processes. And I find it interesting that we’re still not seeing much uptake of these alternative methodologies, beyond the standard orthodoxy of the Lean Startup.

I suspect there is a competitive advantage to be found here, if you could adapt these ideas to your own work.

You just have to figure out how.

Originally published 23 March 2021, last updated 24 March 2021.