I built this when I was working with a growth team that was running 90% of the experiments at the company. As we started running more and more, we realized that there were certain best practices that when followed made the whole process a lot easier and prevented things from falling through the cracks.

My natural impulse is always to codify. As soon as I see best practices emerging, I say, “Let's write these down,” so that everyone has access to this and so we can scale efficiently. As we spun up new growth teams, we could give them a playbook that was tried and tested through all of the team retros and different experiments we had already run.

We needed to make sure that the experience wasn’t doing harm and that nothing broke before we ramped. But as soon as we felt confident that we didn’t break anything, we needed to get up to that max power as quickly as possible, because that's where we got learnings the fastest. Then we could make a decision and move on. We always tried to be in one of those two states — either the initial early testing phase or the max power — rather than hanging out somewhere in the middle.

This is very dependent on the nature of the product. Medium was a consumer product that in many cases was free, so we were freer to experiment on larger populations. Now I’m in a B2B context where our ability to experiment is a bit more constrained.

There was a big category of questions around the timing and sequencing of ramps that we wanted to demystify. This helped teams stage their rollouts to get to statistical power as efficiently as possible.

One important note is that Medium was using a homegrown experimentation platform. That means this best practice is specific to how this particular experimentation platform was constructed. Any experimentation platform has its quirks, so being able to pierce through and understand the major gotchas of your particular tool is the most important takeaway here.

The right number of variants is product-specific. The number of variants we cited here was based on the size of the Medium population; this was the right number of variants for us to get stat sig in a reasonable amount of time.

However, in general, I don't think it's a good practice to run too many variants because it can lead to sloppy thinking and throwing spaghetti at the wall. Limiting the number of variants can be a good forcing function for PMs to be really disciplined and strategic in their upfront thinking.

We decided to have a holdout group at Medium that wasn’t exposed to any marketing emails, so we could see the total impact of marketing emails collectively. It really depends on the culture of the company and how important it is to be able to have a clean read. Honestly, there's no right or wrong answer.


We need to apply a really tight lens of scrutiny for when we think the behavior will change over time. You actually don't need to have a holdout group to look at people's behavior in 2-3 months if you've set up your experimentation platform well. There are times when repeated exposure actually is necessary to understand the impact over time (if it's a core experience, like a homepage, for example). But keeping an experiment live as a holdout group [for growth experiments] is usually unnecessary.

Caitlin Roman

I codified these best practices for my Growth team at Medium to help us run a high volume of experiments smoothly.

Below are some thoughts on setting up experiments - a work in progress, and please feel free to add / comment as appropriate!


For specifics on setting up an experiment and ramping via go/experiments - click here [link redacted].

1) Experimentation is different from a roll-out

: If we do not know whether something will work / whether we want to move forward, we should always experiment. Specifically, we should hold at max power ramp (explained below) for a statistically significant read before rolling out. A rollout is when we ramp straight up to a high percentage (e.g., 95% or 100%) - this is not experimentation.

2) We should aim to run all experiments at maximum statistical power

 there is a PR / other concern to run them at lower power. To get to maximum statistical power, divide 100 by the number of variants (including control). E.g., if you have 3 variants + control, maximum power ramp would be 25% per variant and 25% control.

Product Science can provide an estimate of how long we should plan to hold experiments at the max power ramp to get a statistically significant (stat sig) read.

3) All experiments should run orthogonal to one another

: i.e., they should use unique md5 hash IDs such that the members that fall into variants of one experiment are distributed evenly throughout variants of another experiment. This avoids experiment bias. There are very rare exceptions to this rule and should only happen after talking to Product Science.

Different hash makes the experiment orthogonal.

Exceptions to running experiments in separate buckets using same hash:

If 2 experiments impact the same surface in a way that they cannot overlap and need to be ramped at different times.

 For example, if we’re running 2 experiments whose populations cannot not overlap, and each experiment has a different targeted ramp time. In this case, we would create 2 separate experiments instead of using 1 experiment with 2 experiment buckets, so that each experiment can be developed and ramped independently from the other.

If experiment A impacts only one group of experiment B.

 In this case, we cannot exclude experiment A’s effect on experiment B by comparing control to experiment.

Verify that bigger features work at a small % before moving to maximum statistical power:

 while we want to hold at max power ramp, we should move more slowly for bigger changes to make sure we haven’t broken anything with the new code. For example, we might go to 5% Monday, 10% Tuesday morning, 25% Tuesday afternoon, and then hold at 50% Wednesday morning for the testing period. The engineer who shipped the code is responsible for managing the ramp and monitoring to make sure things are looking good.

: in order for the experiment controller to work correctly, ramp segments at the max power ramp need to be adjacent to one another (e.g., 1-50 variant, 50-100 control). However, if you have multiple variants you will want to ramp with multiple control buckets until you reach max power. For example, if you had a 2-variant test that you wanted to ramp 1% - 10% - 33% per variant, you’d set up the experiment as follows:

: given the guidelines above, we should try to increase our chances that we will get a statistically significant read from the experiments that we run. Therefore, in general we should run no more than 3 variants (+ control) per experiment.

7) Avoid using holdout groups whenever possible

: In particular, a 90% or 95% ramp with holdout group should be used 

 (or never!) since if we want to get an A/B read on an experiment, it should be run at max power ramp. 

Holdout groups should only be used when we have a strong reason to believe that user behavior will change over time, AND we feel it is valuable to capture this signal on behavior change

 (i.e., would we then ramp back down to 0%?). Ideally, experiments should exist only at max power or 100%.

8) Be mindful of how many orthogonal experiments are running on the same surface

: while in theory we can run unlimited orthogonal experiments simultaneously, in practice there may be interaction effects between experiments on the same surface that may impact readouts. Therefore, we should be thoughtful about how many experiments are running simultaneously. Discuss with Product Science if you have concerns.

9)Think about your control group in relation to what you’re trying to measure: 

we can measure both sitewide and triggered impact for any given change.


refers to the specific A/B impact of the test on the population that experienced that change. For example, triggered impact for a change to the post page sign up prompt would compare control vs treatment 

 people that saw the post page prompt and not all logged out visitors, since they did not have the opportunity to experience either treatment or control.


 takes into account the frequency of the event and extrapolates the A/B impact on the entire population. In this example, sitewide impact would include as the control group all logged out visitors, reflecting the fact that while a change to the post page prompt could have a high impact to those seeing the prompt, the overall impact on Medium will be diluted since the majority of visitors do not see this prompt.

This is controlled by the exposure event.

You may want to set up two experiments, one that shows triggered impact and one that shows sitewide impact.

Guidelines for Experiment Launches and Ramp-ups at Medium

Guidelines for Experimentation & Ramps

Sign up to see more