This is an example analysis artifact to better understand the incrementality of the Faire mobile app. 

 to maximize our impact and this is an example of the final output of that process. Each of our analyses are extremely hypothesis driven and we take the time to scope out our hypotheses and the potential impact of validating each hypothesis in a formal scoping doc. You can see the scoping doc we put together for this specific analysis 

The big thing we were trying to understand here was, was this a problem with the app or with the experiment? That’s how we ended up bucketing our original hypotheses as we dove into the analysis.

We always try to be answer-first in our work. We’re known as a team that gets to the answer quickly, and so we’re always experimenting with ways to surface it in even fewer words. The TL;DR box is meant to answer, “If I’m in the elevator with our CEO, what would I want to tell him about this analysis as we’re traveling from floor 1 to floor 3?”

We don't always deep dive on experiments, especially positive ones. But the reason we wanted to deep dive on this one specifically was because it had the potential to kick off a ton of high-value work on the mobile app if it went well. It was foundational to us understanding: is the mobile app as it is today incremental? 

Because of its importance, we ended up road-showing this analysis a fair bit across the org. It remained in Notion, but we shared it across a bunch of different forums and teams to demonstrate how and how not to run experiments that have downstream metrics.

Learning that the issues were due to experiment limitations, rather than the app, was a great learning for the future. We realized it’s really challenging to measure the incremental lift of mobile app downloads on GMV or brand orders because those metrics are so downstream. The lift was always going to be small, and we realized we should be happy that we saw any lift at all.

These next steps weren’t surprising at all because we had been so thorough with our scoping doc. Our analysis plan and metrics were very clearly mapped to individual hypotheses, which meant we knew exactly what we would need to do.

Understanding the original experiment limitations kicked of a whole slew of new onboarding experiments to try to get people to download the mobile app. It ended up being really important to our strategy going forward!

Jolie Kemp Zwick

At Faire, we always take the extra step to synthesize and codify our insights in a digestible way to maximize our impact. This is an example analysis writeup for an analysis to better understand the incrementality of the Faire mobile app.

The purpose of the great app migration was to understand the incrementality of the app by encouraging mobile web users to download the app through multiple different entry points and offering free shipping on a first app order. The core hypothesis this experiment was trying to validate was whether encouraging users to download the mobile app would lead to an increase in brand orders. While we saw a slight increase in brand orders, this lift was not as large as expected. 

This analysis explored a few hypotheses as to why this might be the case.

We always try to be answer-first in our work

1. A lot of things went really well in this experiment 

The experiment successfully caused app adoption without degrading the mobile web experience:

 we saw a 35% relative increase in first time app downloads and a 14% relative increase in app visits with no meaningful degradation of the mobile web funnel

The users in the test and control who downloaded the mobile app behave similarly:

 we see similar app conversion metrics when looking at users who made it to the app in the test and control, despite the fact that there is a difference in intent for these two groups

2. There were a few key factors impacting the effect on downstream metrics

All downstream metrics are extremely diluted:

 given the nature of the experiment, we are taking several conversion haircuts at each step from modal view to app order and therefore are really only evaluating a small percentage of the test population (i.e. the 14% of the total population that successfully downloaded the mobile app)

It takes a long time to see GMV gains that outweigh the upfront investment:

 getting users to download the mobile app is a tall order and it takes some time for the benefit to outweigh the upfront cost

We ran an experiment to encourage mobile web users to download the mobile app through banners and popups with the hypothesis that this would lead to incremental ordering. This hypothesis was proved correct, as we saw a statistically significant X% lift on brand orders / retailer (our primary metric).

The experiment was extremely successful in getting users to download and visit the mobile app. As we can see in this graph below, we see a 35% relative increase in first time app downloads and a 14% relative increase in app visits (both statistically significant):

This is supported by strong engagement with the modal and banner:

When looking at the mobile web funnel, we do see a slight decrease in our test population (statistically significant). However, given that the purpose of the experiment is to move users over to the app, this decrease is not alarming.

It is extremely encouraging to see the funnel metrics for app users in the control and treatment. These users are quite different — the control users got to the app organically while the test users were encouraged to go to the mobile app through many different promotions on mobile web. However, we see that these users are performing very similarly throughout the funnel..

Thinking back to the first graph, we do see a substantial improvement in the % of retailers visiting / downloading the mobile app. However, when thinking in the context of brand orders, our metrics are extremely diluted because we are also evaluating the 72% of exposed users who did not visit the mobile app. Therefore, even if we saw significant gains in the retailers who did the desired action, this all gets watered down by the retailers who didn't.

Additionally, asking a retailer to click on a modal, go to the app store, download the mobile app, open the mobile app, and then eventually place an order is a big ask. We are taking conversion haircuts at each stage of this funnel, causing a hit on the lift in brand orders. Additionally, it takes some time for the benefit to outweigh the upfront cost. As we can see below for retailers who were bucketed at least 60 days ago, the gap in cumulative GMV / retailer continues to widen as they have been exposed for more days.

Figure out more effective ways to migrate users to the app

Now that we are more confident that we should be migrating users to the app, we should continue to figure out more effective ways to get our users there. This will also help us further increase our confidence of the incrementality of the app. One way to do this is choose a more targeted population of users that we believe will download the app. This could help ensure we aren't diluting our experiment metrics to much and really help isolate the impact. Another way we could do this is brainstorming even more forward ways of migrating retailers to the apps.

Now that we have a good understanding that the app is incremental (and will become more incremental over time) we should continue to invest time and resources into the app to make it the best it can be!

Great App Migration Analysis at Faire

Overview

Key Takeaways

Sign up to see more