The impetus for this template was that - at this point - we weren't that sophisticated at doing experiments yet. And so getting multiple sets of eyes (in particular mine or the head of data) on the experiment was important. We wanted to pressure test whether the thing was worth testing, whether the experiment was actually testing the thing they wanted to test, and whether they'd even get any useful data back. Those things are more important for an experiment than they might be for bigger projects (which could be more about the broader customer experience).

With an experiment, since measurement and compounding learning are so important, you have to make sure that you're actually going to get something back.

In particular, we had been noticing some pain around experimentation that resulted in the need for this template. For example:
- Too many things changing or not assigned properly
- Experiments without good hygiene: the right metrics weren't being measured or they weren't even useful metrics

There are just a lot of things that can go wrong in an experiment. So a lot of this is trying to make sure that we're running good experiments, not bad experiments (More on that 

You'll notice this looks pretty similar to a product brief. But let's talk about why they're different.

The goal of the product brief is to evaluate what we should be spending time on.
- What's the customer problem?
- How do we know it's a problem?
- Do we understand this customer problem well enough to approach it?
- Is it a big enough problem?
- Does it have enough business impact?
- What's our initial solution?
- Etc.

A well-written product brief answers that, and the hard work is really getting at the customer problem.

In contrast, on an experiment, it tends to be less important to stack rack or prioritize the things.

The focus is less on the customer problem and the exact solution; instead, the focus is on the setup. It's more important to ask the questions, "How will we know if this worked and what will we learn from it?" How are we going to set this up in a way where we'll actually learn from the data we get back?

The audience for this document was the product lead, the data scientist, and maybe engineering depending on how complex the experiment was.

We would review them in person as part of our product workshop / product reviews. Over time, it became more async, where the product lead would drop it in Slack, and then someone would review it and put a check mark on it.

But in the beginning, it was really about going back and forth on it and having discussion around it.

This was a living-ish document. I've struggled to make product specs and product briefs living documents in the way that everybody would like them to be. People don't update them, the results don't end up in the document and end up somewhere else instead. It's very hard to be like, here is the perfect representation of what happened.

But this document sort of replaces a bunch of different stuff - like PRDs - that would live in a Google Doc black hole.

The other goal of this tool is to get alignment from the people building around what they're testing, why, and what's important. When you're running fast product cycles, you don't tend to go through all the "proper steps" in the most formal way. You're kind of just trying to run at it. Alignment and clarity around what's important, what's not important, what we're testing, and how we're going to measure success means you can make micro decisions along the way.

So now, for example, the designer knows that, okay, we're testing this idea, so let me make sure my design is a big enough change to do that. Right. Now let me make sure that -- as I implement it -- it's actually testing the thing we say it's going to test. Let me make sure I can actually assign the right people into the experiment. So it's an alignment tool as well, in addition to just a pressure testing tool.

Even though this wasn't always a true living document, this was the one big thing we would always make sure to look back at. We wanted to look back at the success criteria and make sure that we were analyzing the things we said we were going to analyze when we built the thing.

I'd say about one in five actually filled this out.

Fareed Mosavat

At Slack, I used this experiment design and documentation template to get the team to openly document and pressure test the goals of any planned experiment, increasing the team’s sophistication around experimentation.

Contact: @pm @design @engineers @datascience

Current status: What % of target audience is this launched to? If not launched, what development stage?

Who was the target audience for this experiment?

Who was exposed, and under what conditions?

Any other useful information (work vs. social, admins vs. all users, plan types, etc)

Pre-launch, include estimates for the audience size.

Post-launch, include actual audience sizes for each variant (available in experiment framework).

Describe the experience for both the control group and any experiment variants. Describe them in enough detail that someone unfamiliar with the feature could effectively understand what you are testing, the experience for customers, and how to interpret data and results.

In the definition stage, user stories or a basic description will suffice.

After launch, please include screenshots and detailed descriptions. This will help the next person reading the doc better understand the state of the product before and after this launch.

What are the key success metrics for your feature? If your hypothesis is correct, what is your best guess for what the impact will be?

Please list out any assumptions you are making and links to the data you used. Usually this should include a baseline funnel, your best guess at how that funnel will change, and an incremental impact.

It’s important to do a strong estimate of the reach for your experiment - how many users will actually see this experience?

How long will this experiment take to converge with powered results or shut down because of no impact?

If you are just doing a launch results analysis, you can skip this section. This is most valuable before you start an experiment.

What did we learn? What are the most important takeaways? Did this match expectations?

What is your recommendation for how to move forward?

What changes were observed across all key metrics?

What is your interpretation of these results?

Were there any problems you encountered that may affect how we understand what happened?

After launch, if the experiment is successful, it’s helpful to frame the topline impact on key metrics, since all users are likely not exposed to the feature. If we ramped this experiment to 100%, how much should we expect the target KPIs to change?

For instance, if an experiment focused on iOS team creators showed a 5% increase in team creation, how many teams per week does this represent? Normally this is represented on a weekly basis.

Post-launch, please include the timeline of the experiment, including all changes to exposure, rollbacks, etc.

Include screenshots from the experiment framework for key metrics, any additional screenshots that are helpful, any supporting dashboards.

Experiment design & documentation template at Slack

Internal Template

[Your Feature Name]

Goals and Hypotheses

Sign up to see more