The biggest challenge teams face with experimentation is that they don’t know what they don’t know. In a B2B context — particularly in the early stages — many companies simply don’t have the volume to even consider running formal experimentation programs. A lot of the guidance in this template is around that strong hygiene that is required for setting up a structured experimentation program, as opposed to throwing stuff at the wall and seeing what sticks.

When you have programs of experimentation that are running at scale, It is inevitably difficult for someone to come in, see it all and make sense of it all. You do generally have to be fairly close to it. So it becomes incredibly important to focus on how you tell the story, how to communicate and evangelize what's what you've been doing to the rest of the organization. One of our big opportunities to do that was a monthly impact and learnings review, where we would surface some of this stuff in aggregate for stakeholders across the company. There's always an opportunity to pull things out at a slightly more aggregate level and chain together learnings.


A bad learning objective is one that is focused more on the outcome we want to achieve, not what we’re trying to learn. You don’t want something like “When this experiment concludes, we hope to have improved the ability of a new user to overcome this point of friction.” That might be a great outcome, but an experiment learning objective should be framed in terms of what evidence you hope to find that can then be applied to generate impact elsewhere.

If you’re in a very mature organization with large volumes of data, you’re going to be more strict in applying these guidelines to your hypothesis generation. For earlier stage companies, you might be a little more lenient and rely on qual more than quant. But the point is you want something there in all cases, just to validate that this isn’t just someone’s great idea; this is actually coming from something we’ve learned.

We explicitly built a page in our growth hub on Notion that helped teams learn how to experiment in the right way. And one of the pages within that was focused on when NOT to experiment. Do you have enough evidence already to say, “Just get on and ship this thing?” Helping teams understand the right and wrong time to apply experimentation was important.

My hope and expectation was that anyone on the team should be capable of owning an experiment end to end, which meant some foundational level of statistics. However, that doesn't mean that we didn't benefit from having someone in decision science supporting all of our growth teams when it came to any questions they had about the stats. The decision science team was a critical support function especially in the first 6-9 months as folks were really becoming more familiar with and experienced in how to design experiments.

The single most important thing we did here was setting a realistic expectation around the volume of wins we would expect. We put a stake in the ground that we should expect roughly around one in five wins, or 20%. If we are seeing a significantly higher win rate than that, then we're probably hedging our bets a little bit. We're not taking risky enough bets. Building on that, we wanted to make sure people understood that every single experiment that you can draw valid learnings from is actually a win, and it will incrementally move you closer towards your ultimate objectives. 

A failure is if the experiment is invalidated in some way. That might be an execution invalidation, meaning you've got the experiment live, but you haven't coordinated well enough with another team who works on that surface area, and they've made a change and there's our data out of the window. Or it might be a design failure, in that you've picked parameters that mean that ultimately the experiment concludes and you haven't learned anything. The important thing is to celebrate valid learnings, even if they don't deliver the the outcomes that the hypothesis estimated.

Ben Williams

I created this growth experiment template to assist in setting up a structured experimentation program, as opposed to throwing stuff at the wall and seeing what sticks.

What do we intend to do, and what are we trying to learn by running this experiment?

describe the experiment in plain language

We want to <do this thing in the product> for <this group of users>. We hope to improve <this metric>, and do no harm to <this metric>. When this experiment concludes, we hope to have learnt <learning objective>*

What is our hypothesis? This should be grounded in some prior data (qualitative and/or quantitative) to maintain experiment quality and avoid reliance on opinion.

Because <some insight/learnings/data/evidence we have> We believe that <doing this thing> Will result in <some change to this metric>*

Remember that great hypotheses are grounded in evidence, not plucked from thin air.

Be specific with the evidence t*o help people understand the why behind the hypothesis.

It’s good practice to link to any research collections, analytics charts or other evidence supporting the hypothesis.*

The observation of the situation (e.g. only 1.8% of users on this screen click the CTA), and

Why we think the test experience will change the observed situation (e.g. user research suggests that the CTA is not noticed, and a competing CTA that is more prominent is clicked by 16.4% of users)*

Because only 1.8% of visitors to the homepage click on the CTA to sign up, and a competing CTA to book a demo is clicked by 16.4% of users, and recent user research suggests the signup CTA is not noticed,

We believe that changing the location of the signup CTA to make it more prominently visible,

Will result in more visitors seeing and interacting with the CTA.

Use this section to elaborate on and/or link to 

Recordings of user interviews or observational studies

Summarise the control group and what they will experience.

Include screenshots or anything valuable to convey the experience.

Summarise the test/treatment group and what they will experience.

We will test our hypothesis by assuming the change has no effect (the null hypothesis) and running an experiment for the target runtime specified below.

If we can measure a statistically significant change (of at least the MDE specified below) in our primary metric, then we reject the null hypothesis and conclude there is an effect.

Use this stat design tool to help identify appropriate parameters

What we will do in certain circumstances.

What level of result is considered success?

Who might the outcome of this experiment be useful or interesting to?

What was the impact on secondary and guardrail metrics?

Ensure that you link to charts/notebooks that document the experiment data.

Growth experiment template from PLGeek

Growth Experiment Template from PLGeek

Summary & Learning Objective

Hypothesis

Sign up to see more