Product Validation Frameworks are Mostly Useless Without Taste

One of the things that I was trying to get at in Product Development as Iterated Taste was this idea that all product validation methodologies consist of two parts: an explicitly followable process, and a tacit component that may be best described as ‘having good product judgment’, or ‘product taste’. Following the explicit process would only do you so good; there’s still a tacit component to learn that is important if you want to be repeatably successful in creating new products.

In the two months since I wrote that post, I’ve had multiple casual conversations with entrepreneurs and product builders about Amazon’s intriguing PR/FAQ process, and what we might learn from it. I found myself saying that focusing on the trappings of the process alone isn’t going to do too much for you; you also have to acquire good product judgment, or run the process with individuals who possess good product judgment. I learnt that I didn’t do a good job of expressing this idea in my pieces — in fact, I realised that I never even articulated the argument; I merely alluded to it in Product Development as Iterated Taste and in my summary of Working Backwards.

I do think this is an important idea to call out, though, because a naive reading of product development methodologies is “ahh, I just have to follow this process slavishly, and I’ll be able to build a product that people want.” Which — as anyone would know if they’ve ever put any of these product development methodologies to practice — is simply not the case.

Learning All the Ways Product Validation Can Fail

I started my first startup at the end of my freshman year of university. It was an ebook publishing software company, and like most first startups, it was a disaster. We didn’t talk to users before launch, I fought excessively with my cofounders, I didn’t understand what it took to build working, usable software, or how to think about tech debt (a good thing, if used strategically!), or how to do user testing, or how to run proper product analytics. It was a great experience, though: I shored up most of those skill gaps in my remaining years at university. I read Eric Ries’s The Lean Startup a couple of months after this experience, and swore I would use it in all my product building attempts going forward.

After graduation I was hired to turn a Vietnamese consulting company into a product business. Before we pivoted into Point of Sales systems, we built a mobile ad-network, three mobile games, and a CRM product for insurance agents, while running an app consulting business at the same time. I made sure to use the Lean Startup methodology for all of them. None of them succeeded:

• The mobile ad-network was a copycat idea of replacing the lock-screen on Android phones, and paying users to look at an ad for a few seconds on every unlock. This had pretty good traction amongst Vietnamese end-consumers, but we couldn’t get ad buyers to bite. (Much later I learnt that incentivised ads — that is, ads that pay for user attention — are the least valuable ad type; the most valuable ads have high-intent, which explains why Google is Google and none of the lock-screen ad companies are anything today).
• The mobile games we built failed to gain traction because game design was difficult, and even in 2015 the gaming market was a knife fight.
• The CRM product for insurance agents failed because it was focused on helping agents track existing clients, and insurance agents weren’t incentivised to check in on existing clients so much as they were incentivised to find new ones. I have more nuanced thoughts on this, but I considered this project the most successful of the set in terms of executing on the Lean Startup methodology: we took no more than three months from project initiation to decision to kill the idea, with plenty of user interaction, a few serious beta testers, and two solid product iterations in between. (And then I sneaked into an internal agent training session at AIA headquarters, and realised that we had built the wrong thing — but that’s a story for another day).

If you zoom out a bit, I think we iterated through most of the failure modes possible when attempting to use the Lean Startup:

1. I didn’t talk to any users (first startup). This is, to be clear, more a failure of not using the Lean Startup, than a failing of the Lean Startup method itself.
2. We built something people ‘wanted’  — as measured by engagement — but couldn’t monetise (the mobile ad network).
3. We couldn’t iterate our way to something compelling (mobile games).
4. And with the insurance CRM we talked to potential users and moved rapidly, but did not understand the incentives that governed their behaviour — in other words, I committed every sin in The Mom Test.

Of course, it’s not clear that these are the right lessons, since building products are a wicked learning domain, and it’s important to have a loose feedback loop when reflecting on past failures in such domains. I continue to update the lessons I’ve learnt from those experiences today. But one thing that’s stuck with me was just how much judgment was required during iterations: I remember thinking that I had to put in way more reps if I wanted to become good.

A few months after we shuttered the CRM app, my boss called me on Skype and said “Hey, let’s do Point of Sales systems. I’ve bought the remains of a POS distributor, and I’ve got one customer lined up.”

I told the team. We said yes.

The rest, as they say, is history — within months we were making more than we had made in a full year of consulting work; when I left, the company was generating millions in annual revenue. I helped build five other products over the course of my tenure, each time iterating closely on customer feedback. All five products were more profitable than anything I’d ever built in the years before.

Taste vs Customer Obsession

What changed? Was I a better product person than I was during the months of the five product experiments? Did I run the Lean Startup methodology better than I had with my previous attempts?

No, I don’t think so. I was marginally better at asking questions (I didn’t egregiously repeat the mistakes from The Mom Test, for instance!) but I don’t think I was that much better at product judgment, nor was I suddenly better at building stuff that people needed.

What changed was the product domain. It was a simpler maze to navigate. As a business mentor put it, years later: “What you’re doing is easy, because you’re making Point of Sale systems for retail customers, who know exactly what they want. But if you try and build something completely new for them, you’ll be struggling with all the same problems, all over again.”

He had a point.

There are, broadly speaking, two kinds of product development processes:

1. You ask customers what problems they have, and build for those stated problems. In some domains this may take the form of asking customers what they want, and then building exactly what they want.
2. Or you don’t ask the customer anything, instead you iterate internally and rely on the taste and judgment of internal product leaders, before you release the product to great success.

Veteran CEO Frank Slootman puts it slightly differently, in Tape Sucks:

Not every startup lends itself to such an intense customer focus: some new ventures are supply-led, not demand-inspired. Consumers could have never asked for an iPod until they actually saw one first. Our business was about incrementally improving an existing process, an example of straight-line innovation. We didn't propose to customers to do away with backup altogether, just the devices that they backed up to. We still worked within the confines of an existing process, which lent itself well to an intense customer focus.

All of which is to say that both are valid product development processes. I’ve come to see customer obsession and taste as two parallel paths to product success. In some domains, you can get there through customer obsession alone — ala the Lean Startup. My previous company was very much like that, as was Slootman’s Data Domain. It was ‘simple’, even if it wasn’t easy. We knew what needed to be built at each step of the way. In other domains — when you’re building TikTok, say, or the first iPhone, the idea maze is a larger mess, so the process is more dependent on judgment and taste.

That taste, of course, is tacit in nature. But people don’t seem to want to talk about that.

The Four Product Methodologies

Once you have this lens of customer-obsession vs taste, the four product methodologies that I covered in my previous post begin to make more sense. You can’t copy what Apple does, or even what Amazon does with its PR/FAQ process, if you don’t already have people who have good product judgment.

To recap, those methodologies are:

1. The Lean Startup — You interview prospective customers, release a ‘minimum viable product’, and iterate rapidly based on feedback.
2. Amazon’s PR/FAQ Process — You iterate on PR/FAQs, which are six-page documents that force team members to focus on the customer. Often, prototype iteration occurs concurrently with PR/FAQ iteration. In a throwaway paragraph in Working Backwards, authors Colin Bryar and Bill Carr write that Amazon has a tier of master evaluators, who have more product taste (or product launch experience) than others; these evaluators are required to sit in on PR/FAQ meetings. In essence, they spread their taste by osmosis.
3. Apple’s Creative Selection — You iterate on demos, and present these demos to a hierarchy of product leaders. First you present to your boss, then your boss presents to their boss, and so on up, with the top level being Steve Jobs himself. You are invited to be an evaluator on these product panels if you have proven yourself to have taste; similarly, if you make a bad judgment in one of these demos, or make a stupid comment, you would not be invited back.
4. Pixar’s Braintrust — Directors iterate on reels — that is, crudely drawn storyboards that are edited together with temporary voices and music — which are then presented to a core group of experienced storytellers. The storytellers debug the script together, and then directors go off and iterate on the feedback. Admission into Pixar’s Braintrust is — like Apple’s demo panels — by invitation only. You have to prove your storytelling judgment before you’re allowed in.

When viewed through this lens, the majority of alternative product development methods seem to demand that you have taste in order to succeed. And so if you attempt to copy Amazon’s PR/FAQ process without having someone with product taste in the room, it’s likely that you would fail spectacularly — as Amazon did when they ran the process for the very first time.

In practice, many products require a mix of both taste and customer obsession. Hang around product people long enough, and you’ll realise there is a subtle difference in the way experienced product people run iterations, compared to those who are more junior (or who have never built products of their own). They know how much to pay attention to customers, and how much to ignore them. In the absence of data, they make gut decisions — and their gut decisions tend to be right.

To repeat my assertion at the start of this essay: all product validation methodologies consist of two parts: an explicitly followable process, and a tacit component that may be best described as ‘having good product judgment’, or ‘product taste’. Following the explicit process would only do you so good. In certain domains, success depends just as much on the tacit component, which is harder to articulate, and therefore harder to get.

And so the next question you might ask becomes: what can you do to acquire better product judgment? How might you recognise when someone has it? How might you develop taste?

These are good questions, but the answers that I’ve uncovered so far are incomplete and unsatisfactory and long. I’ll leave them for another day.

Read the follow-up (members only): This is What Product Taste Looks Like.

Originally published 29 June 2021, last updated 07 July 2021.