We staffed a Growth pod once at MasterClass whose first two experiments were both ~10% winners but didn’t hit statistical significance. It turned out the conversion rate for the target metric was far too low to detect anything other than a 40% winner. ​​
​​Ever since, we’ve started running a power analysis quarterly during planning, to make sure teams targeted realistic metrics and had a reasonable chance of success.


 that was custom-tailored for identifying what sizes of swings various teams would be to get to stat-sig for.


Being able to identify cases when a team doesn’t quite have the numbers to run experiments as early as possible in the planning process has prevented lots of wasted effort. In this case, the Referral team did not quite have the numbers to run in a "lots of experiments" style; knowing this allows us to adjust the team's work style to more 0-1, "let's just ship lots of things we believe in and look at pre/post quarterly" style.

Alexey Komissarouk

I developed this calculator and ran quarterly power analyses at MasterClass to make sure the right growth teams were taking on the right projects.

This Q3, we are set to staff pods around Acquisition, Activation, and Referrals in Q3. 

This doc proposes metrics for each of the planned Growth Pods. Run a power analysis to verify that each team’s goals are quantifiable and within reach.

We’re fine to run a team focused on checkout experiments.
If we target all the way through to purchase, we’re probably going to want to run a “big swing” type team where the PM takes the lead in prioritizing experiments. If we target mid-funnel, “checkout started” type experiments, we can probably empower the engineers to run these directly & launch a ton of them.

Our original plan to target 30 Day actives is feasible, but suffers from two sides: first, we’re stuck ~30 days to get results, and second, we’d need to take relatively big swings. If we instead switch to 7 day actives, we have significantly more experiment ability to detect smaller changes faster, especially if we are comfortable including free trial users.

Given the numbers, the original plan of targeting weekly active users sending referral codes all the way through to recipient activation is not feasible. We explored limiting the audience to new users, since that is a prime referral time, but that actually does more harm than good. If we still want to go forward with this team, we’ll have to target a mid-funnel metric like a referral being sent & only focus on a particularly active segment such as new users. However, even then we’d have to listen our statistical significance level of rigor & extend experiment runtime to 3 weeks to detect even a size-able 10% win.

Growth Team Swing Size Calculator at MasterClass

Q3 Growth Pod Power Analysis

Motivation

Potential Metrics

Recommendations

🟢 Acquisition Pod

🟢 Activation Pod

Sign up to see more