Before we built this, we had this team called the Fox team whose job was protecting engineers from ad hoc work, so they would take on the bug backlog. It was rotational, and every engineer did a cycle. But we were moving into a world with durable product teams owning specific surface areas, so we wanted to move bug ownership back to those teams.

The problem was that didn't have a standardized set of criteria when we moved to that mode. It was left up to every team individually, and we were finding that not enough bugs were being addressed and we couldn't track engineering excellence. So we wanted to give people a framework for objectively measuring this stuff. It represented a step-function change in the company, both culturally and measurement-wise.

At almost every company, you'll have to define something like this at a certain level of scale. We're doing this right now at ResortPass — we're processing a lot more money than we were a year or two ago and the systems we had in place no longer make sense. You have to go through this process of retraining your organization eventually if you survive long enough.

The color coding showed us what was super severe, but it also showed us what didn't matter. It gave teams a clear rubric to say "This is a trivial thing, and we aren't going to fix it." For example, if something affects $500 of GMV, that's basically a rounding error. Patreon processes $1.5 billion GMV a year. So this was really freeing for people.

An incident is the most severe P0 — that's like the site is down, fail whale is happening, get everybody on board at 3 AM on the weekend. Not all P0s are incidents, but there's overlap.

Before we introduced this, we had plenty of things that people wanted to be P0s but clearly weren't. That was one of the biggest benefits of this: we could clearly show that not every bug is critically important, and it was more important to finish roadmap work than drop everything and fix it.

Beyond P1, there wasn't a huge chance that stuff would get worked on immediately. P2 was maybe inside the quarter. P3 usually got triaged and marked as "not going to fix," or maybe we'd get to them in a polish week. At ResortPass we call this the "happiness backlog." It's just small UX nits that make internal people really happy but don't make a difference on any metrics.

Adam Fishman

When Patreon hit a new level of scale, I needed to create a new rubric — and retrain the organization — for prioritizing and triaging bugs.

A measure of how impaired the product experience is

Trivial — A cosmetic issue and/or inconvenience with a simple workaround available

Minor — An error that impacts product functionality, but not Creator earnings

Major — An error that may impact Creator earnings

Critical — The product is unusable and/or data loss occurs

A measure of the business impact to Creators (and Patreon) defined as "TMV exposure" (users weighted by TMV) — For example: an issue impacting one $10k Creator is equal to an issue impacting one thousand $10 patrons

Directly affects a minority of users || $500 TMV exposure by Patrons or Creators

Directly affects a small amount of users || $500 < x $5,000 TMV exposure by Patrons or Creators

Directly affects a significant amount of users || $5,000 < x < $250,000 TMV exposure by Patrons or Creators

Directly affects most, or all, users || $250,000 TMV exposure by Patrons or Creators

The color coding showed us what was super severe, but it also showed us what didn't matter.

Bug SLA commences at point of filing using calendar days rather than business days

Bug Priorities and SLAs at Patreon

Bug Priorities & SLAs

Prioritization Rubric (Severity x Prevalence)

Severity

Prevalence

Sign up to see more