One of the things I really appreciated about Masterclass was that PMs were really low ego. They would manually QA every experiment, which might be "beneath them" at other companies. But at Masterclass, there was this great culture where PMs were the ultimate owners of their experiments, so we built up a really strong sense of rigor around what are all the things we need to check before this goes out.

When I started consulting, I saw lots of experiments have adverse effects on the product, and I thought, "This never would have launched at Masterclass." I realized what a high bar their PMs had set.

This is a starter template, but ultimately an experiment launch checklist should be specific to your company, because the subtleties and edge cases worth checking will be different each time. To start building this out, I recommend checking your postmortems. If you're running effective postmortems, you'll have a documented list of "the check we didn't run that would have caught this." That's a good place to look as you develop your checklist.


Here's a great example: At Opendoor, we'd send out emails to customers letting them know how much houses in their neighborhood had recently sold for. But in Texas, it's actually illegal to share the sale price of a home. A new PM who wants to test those emails might not know that, so if I were a seasoned PM at Opendoor, I'd document something like "Always check Texas" to ensure that the template doesn't crash or violate any laws.

These are the ones that most often go wrong. If you're at a US-based company, your PMs or engineers are likely in the US, they're likely on a desktop, they're logged in, etc. Depending on the nature of your product, the world may be very different, so you just need to know what are all the important ways that it could be different?

Alexey Komissarouk

In this sample experiment launch checklist, I lay out how the team can avoid common pitfalls before even starting their experiment.

Before launching an experiment in production,

 to avoid common classes of failure. Here is a sample checklist (feel free to 

Before Launching an Experiment into Production (often before merging the PR)

By somebody with context on the experiment but 

the engineer who wrote the code (due to blind spots). The Product Manager is an ideal person to run through, but Design, QA, or an Engineering Manager are other good options.

Is there an actual, well-defined set of test steps (example)? 

If not, have the PM or Engineering Lead write these up

Execute the test steps for each variant & ensure they match expectations

Does the visual design match the original designs?

If not, has the designer or PM approved the deviations?

Does the appropriate “Experiment Exposure” analytics event fire for each of the variants?

Do the expected Analytics Events still fire in each of the variants?

If there is not an explicit spec defining which events must fire in this surface area, you can instead compare the events firing in Control vs the Variants

For each exclusion condition that makes customers ineligible for the experiment,

Is their experience equivalent to control?

Does the appropriate “Experiment Exposure” analytics not fire?

Note: not all of these apply to every experiment surface area; please adjust as necessary

Is the visual language & app behavior consistent across each variant with...

expect: see big blue button that says “get started”

expect: now on page 1 of checkout, at /signup

expect: see big red button that says “start now”

Experiment Launch Checklist from Alexey Komissarouk

Experiment Launch Checklist

Motivation

Conditions

Checklist

Part 1: Tablestakes

Sign up to see more