“Don’t build anything unless you get validation first. Build the button, but not the feature. Track the results.” – Sebastien Brault, Serial Entrepreneur

A common way to pre-test features is to set up a Fake Door feature tests.

The general idea of a Fake Door experiment is to pretend to provide a product, feature, or service. For example, adding a button or menu item for the feature that you’re thinking of building, without actually building it.

When users click the button, they’re taken to a page explaining the experiment and how they can contribute by filling out a survey or taking part in an interview.

The Issue With Feature Tests

Fake Door demand tests can tell you if the idea sparks the interest of users. But there are a few challenges with this technique:

  1. Clicking on a link or a button doesn’t actually mean that the feature will be perceived as valuable. Curiosity clicks are unavoidable.
  2. The button or link placement will affect test results.
  3. Button or link labels can set the wrong expectations.

What a Fake Door demand test (feature tests) can reveal is whether there is interest in trying a feature, which is only one part of the picture.

What You Can Do Instead

A perhaps more valuable way to learn about demand is to reverse this test, hiding or removing existing product features. You can design an A/B test that hides the feature for some users. If the feature is valued, its absence will impact usage. You will see an increase in support tickets and social media feedback. Those can be great indicators that the feature was valuable.

A less disruptive approach would be to ask the main question of the product-market fit survey after a specific feature gets used (“How would you feel if you could no longer use [ Feature ]?”).

As with the general product-market fit survey, the idea is that if at least 40% of respondents state that they would be “Very disappointed” if they could no longer use the feature, you can infer that this feature has market fit.

Following up with an open-ended question (“What is the main reason for your answer?”) can help reveal gaps. This will help you understand why the feature is valuable to some users, and not to others.

Surveys can help you make better product decisions, but don’t surrender decision-making to a survey. They’re simply there to help guide product decisions.

Sometimes reduction is the best way to make the existing value in a product more noticeable and effective. Don’t be afraid to remove features when they don’t move the needle.

Killing Features

Katryna Balboni of Appcues recommends making killing features a regular part of your product strategy while Mind the Product co-founder Janna Bastow, recommends creating a ‘kill list’ of features.

Weigh the pros and cons. Ultimately, as Dave McClure says: “When you kill the wrong one, people will make noise and you’ll be clued in to what actually adds value.”

– –

This post in an excerpt from Solving Product. If you enjoyed the content, you'll love the new book. You can download the first 3 chapters here →.

Categories: Common Challenges