This is an example of how we scoped out an analysis to better understand the incrementality of the Faire mobile app. At Faire, we start each analysis with a scoping doc to maximize our impact. To learn more about Faire’s scoping process, see 

We only do hypothesis-driven analysis at Faire. Otherwise, you end up with a bunch of fun facts and additional questions rather than actionable insights. A good hypothesis is specific and targeted, and not afraid to be wrong. A bad hypothesis (or a non-hypothesis) is very vague. For example, if we’re trying to figure out which user actions are the highest value, a bad hypothesis is “Certain actions are more high value than others.” A strong hypothesis for that question might be “Search is the most valuable action a retailer can take.” It’s very targeted, and it can be proven or disproven.

In this case, the PM and I pressure-tested each of these hypotheses and felt confident that each one had a clear next step. I think that’s a sign of a good scoping doc, when each hypothesis has been individually pressure tested.

It’s actually good to start talking about specific solutions during scoping because you want to determine what next steps are in and out of scope. There are certain hypotheses that you could validate or invalidate, but there's no way you would actually be able to solve them. Understanding what is on the table to fix is super helpful when determining which hypotheses to prioritize in the analysis. 

You don't want to get too specific on the solution before you know the answer (e.g. getting into the weeds on how to redesign a modal in our product). However, knowing that we have the appetite and bandwidth to redesign a modal is good to establish during the scoping stage.

When outlining the analysis plan, it is best to organize this section by hypothesis to truly pressure test whether this set of metrics will be sufficient to validate / invalidate the hypothesis at hand.

Jolie Kemp Zwick

At Faire, we always take the time to scope out each analysis to improve the effectiveness of our analyses. This is an example scoping doc for an analysis to better understand the incrementality of the Faire mobile app.

Deep Dive Scoping Doc: Great App Migration at Faire

The purpose of the great app migration was to understand the incrementality of the app by encouraging mobile web users to download the app through multiple different entry points. The core hypotheses this experiment was trying to validate was whether encouraging users to download the mobile app would lead to an increase in brand orders. While we saw a slight increase in brand orders, this increase was not as large as expected. This analysis aims to investigate a few hypotheses as to why this may be the case.

The experiment did cause enough users to download the mobile app

The experiment degraded the mobile web experience, causing drop off

The mobile app is not impressive enough to cause a change in behavior once users download it

It takes time for the benefit of the mobile app to outweigh the upfront cost of downloading it and the experiment will have a bigger lift than currently measured

If hypotheses 1 is true, we know it is not about the effectiveness of the mobile app itself, but rather the way we are encouraging users to download the app. 

, we will brainstorm more effective ways to encourage users to download the mobile app.

If hypotheses 2 is true, we might have gone too far with how invasive the app promotions are in the mobile web experience. If this is the case, we will brainstorm more subtle ways to encourage users to download the mobile app.

If hypotheses 3 is true, this could indicate that we will have work to do on our mobile app experience and making sure it a superior experience to mobile web. If this is the case, it might be too early to encourage users to download the mobile app.

If hypothesis 4 is true (and hypotheses 1-3 aren’t true), this could indicate that while we need to adjust the observation window for our experiment, everything is working as we originally hypothesized and we should continue investing in our mobile app experience.

% of users downloading the mobile app after exposure

% of users active on the mobile app after exposure

mobile web banner / modal engagement (click the CTA, remind me later, close)

mobile web order conversion funnel (% add to cart, begin checkout, complete checkout)

% of retailers dropping off immediately after seeing mobile app promotion

app order conversion funnel (% add to cart, begin checkout, complete checkout)

% of retailers downloading the mobile app and then placing an order on mobile web or desktop

Great App Migration Deep Dive Scoping at Faire

Deep Dive Scoping Doc: Great App Migration at Faire

Context

Hypotheses

Impact

Sign up to see more