As the PM leading the Consumer Growth and Monetization team at MasterClass, it was my job to gather data and input from partners, figure out what our experiment portfolio would look like for the quarter and socialize it with teams. We were working on the acquisition funnel at the time, and there was a slew of qualitative inputs from user discovery about parts of the value prop that weren’t clear to users, and quantitative data on where users were dropping off in the funnel. We needed a way to take in all of this information and prioritize which experiments we should run, when, and on which surfaces.

This sheet provides a template for how to do exactly that. It’s a framework to determine how many experiments we should run on each surface, given the bandwidth we have for the quarter. So if we’ve determined on a few strategic rocks and decided we can run ~20 pebble-sized experiments this quarter, we might use this data to decide “Let’s do 10 experiments targeting the Home page, 2 on the pricing and payment flow, 3 on the recommendations flow, etc.

Figure out the best way to break up your funnel depending on your team's focus and partnership with other teams.

Because this was such a critical input to our roadmap and experiment prioritization, I would link it in every single communication so the team could see proof of rationale. It’s helpful for getting teams on board with your quarterly roadmap and convincing others why your portfolio looks the way it does.


This template helped us map out the funnel surface by surface, then map out how many views each surface got, and then map out how many people who signed up for MasterClass viewed this surface. This data point is our estimation of the contribution of that surface toward a user signing up for MasterClass. It's a proxy, not a perfect metric.


In general, it's best to revisit and update this data quarterly so that you can prioritize accurately. However, there have definitely been quarters where we haven't had time to update it because we've built up enough intuition about the surfaces over time, and the data is still directionally correct. It doesn't matter whether the conversion rate is 28% versus 30% for example — this process is more about driving comparison between different parts of the funnel and making prioritization decisions.


Minimum detectable effect (MDE) is a calculation that estimates the smallest improvement you are willing to be able to detect. The higher the conversion rate is, the easier it is to detect movements in it. So if the conversion rate is 0.01%, it's really hard to move it; but if the conversion rate is 10%, the sample size needed to detect an improvement is lower, so you can detect smaller improvements too. But if conversion rate is low, you'll need a very large sample size to detect small improvements through A/B testing. So you likely only want to take big swings in your experimentation designs on those surfaces.

This sheet (and the MDE column in particular) definitely helped us rule out a few surfaces where the smallest change that we could detect over several weeks with full traffic was close to 15%. We didn’t prioritize those surfaces as much because we knew we'd have to run those experiments for way too long for a reasonable probability of moving our success metric.

The traffic (views), conversion rate and MDE together determine how long an experiment would have to run to reach statistical significance. From there, you can quickly get a sense of your portfolio. Until this point in the template, you're assuming all the experiments share the same (full) traffic. Now this calculation can now help you set traffic allocations too.

For example, the main landing page or Home Page typically falls into the category of high number of views/low conversion rate. Another surface lower in the funnel might have fewer views but a really high conversion rate. If it's going to take four weeks to get statistical significance on one surface versus one week on another surface, that's a key input into your experiment prioritization and scheduling.

Now you can start thinking about allocating your traffic across experiments or the potential for experiment overlap when experiments are sharing traffic. For experiments running for the same population of users (sharing traffic), you don't want to be running more than 1 experiment on the same surface / part of the user flow, but you also don't want to run several experiments on different parts of the funnel at the same time. Experiments can interact with each other even if they're not touching the same surface. We planned around this as much as possible. But as a growth team at a startup, fast execution and changing plans to react to learnings is key. We lovingly called this "experiment tetris".


In the actual artifact I use at MasterClass, there are a bunch of other sheets in this document that I’ve removed here; they contain the lists of experiments that we actually run based on the first sheet.

In hindsight, investing in automated processes — even if just simple automations & templates in google sheets — is so worth it to up-level the team and save manual ad hoc calculations.

Jisha Kambo

I used this template to generate our quarterly experiment portfolio to maximize the success of our bets.

Growth Experimentation Portfolio Strategy Template at MasterClass

Sign up to see more