Adam on how the Patreon team used this experiment template as a discussion doc to drive alignment among Product, Engineering, and Data Science

It can be easy to get caught up on vanity metrics or things that are easy to measure, instead of the metrics that really matter. For instance, teams can obsess over driving account signups, but if they don't lead to more revenue, what's the point? Push for deeper thinking if it's not clear how the experiment's results metric will really impact the business.

Not everything needs to be run as an experiment. A lot of product teams over-rely on experiments, which can actually be quite costly. Even simple experiments create complexity and overhead. If you're going to ship a change regardless of the experiment results, save yourself the complexity and skip the experiment.

For more on when not to run an AB test, check out 

When teams have problems with their experiments (e.g., confusing or misleading results, bad user experiences when the experiment is live), it's likely because they didn't clearly think through one of these five components of experiment design. It's not a failed experiment if the treatment underperforms the control. It 

 a failed experiment if you can't learn something reliable from it due to poor design or implementation. We drilled this into people's heads at Patreon.

When thinking about which platforms you'll run experiments on, it's important to think through how a platform's users represent your broader user base. For instance, you might see more new users on mobile web but higher-intent returning users on your apps. Alternately, you may see meaningful demographic differences based on Android vs iOS (e.g., income, age, locale, language). In some cases you'll be able to test on one platform and then assume results hold for the others; in other cases, the results may look really different, or users will have a broken experience if they go from one platform to another. This was the big thing for us: we needed to make sure the experience would be consistent across devices and that had to do with the randomization unit above. We 

 kept experience consistent across all device types so this was more of a reminder to do that.

Teams can go really wrong when they pick the wrong assignment criteria. If you pick a point that is too high up in your funnel (e.g., on session start or on log in), you may end up getting a lot of bad data from assigned users who never actually encounter your experiment. If your experiment is on the payment page, why would you include users who never encounter the payment page? This was also critical for us when QAing an experiment.

Length of conversion window can be tricky for some products. If you have a product where a user typically requires multiple sessions or an extended period of time to convert, make sure you're running your test and gathering data long enough to actually capture those conversions, versus just looking at conversions in the first session.

It's important to focus on the most minimal experiment for a few reasons. First, you're able to move faster (and then run more experiments and learn more things) if you keep your experiments limited in scope. Second, your learnings will be clearer if you keep your experiments simple; changing too many variables will make it hard for you to learn what in particular caused a change in behavior.

That said, sometimes experiments have to be large and complex. If you need to improve your onboarding flow, it could take months to test every step of the flow in isolation, and you may miss how steps interact with each other. If you only test part of a flow, you may not be able to see changes to the behaviors that really matter. In those cases, sometimes you have to go big.

In addition to providing proper instruction to the folks implementing the experiment, including sketches and screenshots will help you keep a historical record that you can refer to over time. When your org runs tons of experiments, it's easy to forget what you've tested in the past. Be disciplined about capturing static screenshots and wireframes in docs like this. This will also make it easier for new teammates to onboard in the future and get up to speed.

This links to an internal tool we built that calculates runtime based on our testing framework. It determines how long a test will need to take to show minimum lift with typical site/app traffic.

Depending on your product, seasonality may affect when you want to run an experiment. For instance, an e-comm site probably doesn't want to run a high-risk experiment on Black Friday or Cyber Monday. Alternately, holidays may cause dips in traffic that would require you to run your experiment for longer. In the case of Patreon we had big influxes of traffic on “payment day / week") so certain tests we didn’t release close to the end/start of the month.

In B2B2C products and marketplaces, it's really easy to run into situations where experiments on different user bases interact. Be aware of how experiments on different parts of the product or different parts of user flows can affect each other. If you're not careful, you can end up drawing wildly incorrect conclusions about whether an experiment succeeded or failed due to something another product team was doing. This was a big concern at Patreon because we had Creator teams and Patron teams and sometimes those had surface areas that could interact with each other.

Since Patreon is a B2B2C SaaS tool with Patrons and Creators (similar to Shopify or Eventbrite), we always needed to be thoughtful about how experiments would affect marketplace dynamics. If a change negatively affects Patron conversion rate, that becomes an issue for the Creator team, and our Sales, Community, and Customer Support teams would hear about it.

It's important to put a stake in the ground before you run the experiment about how you will make decisions based on the experiment results. There are some changes that you'll want to ship even if they're metrics-neutral or slightly negative. Other changes will need to be a big win to warrant shipping because of how they impact the product. Articulating how you will make the ship-no-ship decision in advance helps drive alignment. You want to have the hard conversations about whether you'd be willing to ship something to 100% (or shut it down) before the team feels really attached to something they worked hard to build.

This is a useful question to ask since teams may have parts of their roadmap that are contingent on an experiment's results. If one experiment informs subsequent parts of the roadmap, and that experiment fails or gets delayed, then the team will likely have to have conversations with leadership about changing priorities or timelines. This question also helps create visibility among teams in case one team has a dependency on another team's experiments, or a team's experiment creates work for another team (e.g., Platforms, Customer Support).

This Slack channel has been really helpful in creating visibility around experiments across the product team. We were a big feedback culture so we wanted to have public accountability and input. All PMs and analysts are in this channel, so it has helped prevent situations where people have been caught off guard by experiments launching. It has also encouraged people on other product teams to ask questions and share learnings with each other sooner in the process. Sometimes PMs don't have opportunities to work closely with other PMs, so this channel has helped create opportunities for them to learn from each other.

Sometimes teams get so caught up in QA'ing user flows that they forget to QA logging. Then, days after the experiment has gone live, they realize they're missing a key event and have to shut down and restart the experiment. This is super frustrating for the team, so that's why we make it a point to call this out in the checklist.

Adam Fishman

My team at Patreon used this doc to help product teams plan experiments more thoroughly and avoid common testing mishaps. It also served as a discussion doc and alignment vehicle for product, engineering, and data science.

Fill out the sections below at the start of experiment planning.

We plan to change [this part] of the product by altering [this property of it] for [this subset of users]. Our goal is to improve [

], without [hurting this other metric that we also care about].

What is the problem we are trying to address with this experiment?

Example: We have seen pledge conversion hold stagnant for the last 3 months, we are looking to drive up conversion in order to increase creators’ earnings.

What underlying hypotheses about user behavior support these product changes?

We believe that [this thing] will have an impact on [the metric] because of [a belief which was informed by [this briefly presented or linked evidence]]. We will know whether our hypothesis was right when we observe [this change in a metric], and our hypothesis will be refuted if we observe [this other change].

State what we are Learning from this experiment

Why should these product changes be run as an experiment?

🔲 We need to know the impact with precision 

🔲 This change has a potential downside 

🔲 What we learn from this change will determine/impact future product decisions 

Are there past results that informed this proposal?

Do not continue to fill out this doc until you have completed the above section and gotten tripod alignment (see checklist below)

Make sure the sections below are filled out before the experiment is launched.

Platforms

Assignment Criteria

Length of conversion window

 that can validate our hypothesis (i.e. sufficient test) involves changing [this part] of the product in [these ways]. We are also interested in changing [this other thing], but will save it for a follow-up based on learnings from this experiment.

[Here is the link to our sketches/wireframes/screenshots/PR]

The minimum lift we want to detect is <X%>, which corresponds to an increase in the response metric from <Y%> to <Z%>. (There is a tradeoff between precision and runtime - to detect a smaller lift will require a longer runtime.)

How long will the test take to detect the lift at the planned variant split? (

. On high-stake, complicated experiments, you can also simulate the experiment.)

This test should need to run for Z days. We think that Y users/devices/creators/dollars will be exposed.

[Screenshot of the runtime calculator here]

Are there other experiments that may interact with it?

This experiment will run from YYYYMMDD to YYYYMMDD. It needs to run after this other related experiment.

Could this experiment have adverse effects that we’re not measuring directly? 

Does this require input from creator-facing teams?

We should remember to segment by X to properly understand results. We should check in with team Y while implementing this. (see +Creator-Facing Project Checklist (Growth/Creator Acq) )

What are the conditions under which we will ship or not ship this change?

We will ship the Puppy variant if it has a statistically significant lift over the control group. Otherwise it is not worth the additional product complexity brought about by the puppies.

Are there followups that depend on the results of this experiment?

If the puppy variant is successful, we will explore more experiments on bones. If the puppy variant is flat, we will shift our focus to kittens.

What were the results of this experiment? Which version won, and by how much?

What is the final recommendation and conclusion? Any followups needed?

Metrics assessment - were any additional metrics impacted by this change? 

You can make a version of this metrics template to monitor.

Before you start implementing an experiment (PM, DS)

🔲 Fill out experiment proposal section (PM). You may want to check in with the research team (Annie or Amanda) to see if there’s past research that could inform this experiment. Link to scoping the purpose of an experiment.

🔲 Align with tripod on why this should be run as an experiment and that it is feasible to build as an experiment(PM)

🔲 Fill out experiment plan section (PM)

     🔲 Consult with DS & Research on the eligibility and response metric definition

     🔲 Create an amplitude dashboard with baseline conversion for the response metric. You can use the template.

     🔲 Add a screenshot of the runtime calculator based on amplitude baseline

🔲 Share concept and experiment doc in 

. Add experiment to go/experiment-tracking , where you can also check for interactions with other teams’ experiments (PM).

🔲 Get DS sign-off on concept and check with product marketing, CH, and/or creator-facing-teams on experiment risks (PM). See this guide to cross-functional experiments for a list of potential stakeholders.

     🔲 If your experiment is high user impact, you should let the content support team know about the experiment and the potential implications, risks, or users questions by submitting a content request at go/contentsupport

     🔲 QA logging for the response metric

     🔲 Make a version of the Amplitude experiment tracking template for the specific experiment (Eng or PM)

🔲 Update experiment doc with final product screenshots/gifs

🔲 Update experiment doc with start and end dates; set slack reminder for end date

🔲 Activate the experiment (go live!) by creating a version

🔲 Notify #community-happiness and tag @datascience with a screenshot and description of experiment once the experiment is live

🔲 Communicate timeline for when the analysis will be complete to the PM (DS)

🔲 Analyze results and share with team (DS)

🔲 Complete experiment doc with results (PM)

🔲 Share what was learned alongside the rollout decisions to #experiments and possibly with #analytics (PM)

🔲 If the experiment was particularly fascinating, buggy, disastrous, amazing - schedule time to bring insights to #product-brain and to Experiment Traffic Control (ping #experiments).

Experiment Doc Template at Patreon

Experiment Doc Template

Experiment Proposal Section

Experiment Summary

Experiment Owners

Sign up to see more