Great App Migration Analysis at Faire
The purpose of the great app migration was to understand the incrementality of the app by encouraging mobile web users to download the app through multiple different entry points and offering free shipping on a first app order. The core hypothesis this experiment was trying to validate was whether encouraging users to download the mobile app would lead to an increase in brand orders. While we saw a slight increase in brand orders, this lift was not as large as expected. This analysis explored a few hypotheses as to why this might be the case.
1. A lot of things went really well in this experiment
- The experiment successfully caused app adoption without degrading the mobile web experience: we saw a 35% relative increase in first time app downloads and a 14% relative increase in app visits with no meaningful degradation of the mobile web funnel
- The users in the test and control who downloaded the mobile app behave similarly: we see similar app conversion metrics when looking at users who made it to the app in the test and control, despite the fact that there is a difference in intent for these two groups
2. There were a few key factors impacting the effect on downstream metrics
- All downstream metrics are extremely diluted: given the nature of the experiment, we are taking several conversion haircuts at each step from modal view to app order and therefore are really only evaluating a small percentage of the test population (i.e. the 14% of the total population that successfully downloaded the mobile app)
- It takes a long time to see GMV gains that outweigh the upfront investment: getting users to download the mobile app is a tall order and it takes some time for the benefit to outweigh the upfront cost