This process emerged through many feature launches and retros. We’d see some features go well, but other times another team would uncover the feature in production and have a bunch of questions about why it worked that way. My engineering partner and I focused on clarifying who was involved and who was responsible for what during each stage of the product development process. These roles and processes gave us the greatest chance of a smooth launch, with all the right teams being notified and the feature being verified.

By the end of my time at Medium, our team had really got into a groove - we were shipping high quality product consistently, and that was in large part due to our process.

Determining a big vs. small project required a long academic debate, and in the end, there's more gray area than binary black and white. That's why we included lots of examples here. The most significant difference is whether the feature is complex enough to require a bug bash. We wanted to avoid the perception that every single thing (even just a copy change on a banner) required a full-scale bug bash.


The engineering lead would schedule the bug bash and set the stage for it, making it clear how we'd test, the different flows we'd need to test, etc. Often, we’d create a Google spreadsheet with different tabs for each flow, and we’d assign some individuals to each flow and others to just playing with the feature overall. A typical bug bash would be about 30 minutes together in a room (back when we were all working in person). We’d enter the bugs into the spreadsheet live for visibility and transfer them into JIRA tickets after the bug bash.

We had priorities P0 through P3. P0 was launch-blocking, and P1 might block a rollout greater than 5%, for example, depending on the feature. The PM would define what each "P"-level meant at the end of the bug bash.

The first three steps in the announcement process are owned by engineering because they're more tactical. The #visible-changes channel, for example, is basically a log of everything that has gone out. So the best person to update that channel is the engineer because they’re the one who literally pushed the code live. Rather than waiting for the PM to learn about the change and have them update the team, we’d get an immediate update. There is no commentary in that channel. The PM is responsible for announcing the feature later, with commentary and a link to the product brief, in the #prod-announce channel.


Establishing this connection between engineering and product science was the most challenging part of setting up this process. People weren't actively pushing back, but the ingrained behavior wasn't there. I think sometimes there's an unfortunate cultural situation where engineers don't take responsibility for data quality, and it's considered P1 or P2, not a core part of the product. There's sometimes a feeling that data isn't a first-class citizen, and that fundamental philosophical disconnect can cause a lack of communication.

This channel is where the Customer Support team gathered. They'd likely see the feature launch announcement in #prod-announce, but this step was actually going to the support team and checking in to see if we were seeing any user reactions or feedback.


Caitlin Roman

There may be no such thing as a “perfect” launch, but this process helped my teams at Medium consistently ship high quality experiments.

As an Engineer, you’ve built something cool! It works and it looks great and you’re ready to ramp. BUT WAIT. Have you verified it with Design and Product? You should and how you do so will depend on the complexity of the project:

A big project is something where there is a higher risk of miscommunication or things falling into the crack. It will have some of the following characteristics:

Multiple engineers or multiple designers working on it.

Multiple iterations of product/design/copy.

Significant changes to both client and server.

It touches complex business logic that has high impact on our users (SUSI, Memberships, Payments)

 Design Quality check-ins During development: (Eng/Design) Schedule time to check for design quality issues.

: (Eng) Schedule a time for stakeholders (and optionally the entire team) to test the flows. Please prepare a testing environment for the bug bash. This may include turning on variants for MEDIUM_STAFF or loading up test devices. Use the first 5 minutes of the bug bash to go through the Launch Checklist. Track bugs in “Bugs found” tab of the project landing page and assign ownership. The PM will categorize them as 

go/launch for a template for the Bug Bash.


 Verify with design and product that the bugs are addressed, preferably in aggregate. All P0 bugs must be addressed before external ramp begins!

A small project is something that is relatively self contained, usually a smaller experiment. Small projects will generally lack the characteristics of big projects, listed above. Examples of ARC Small Projects include 

Upgrade in Nav experiment, Exp Dashboard UI updates, Direct sign-in experiment

 Design Quality check-ins During development: (Eng/Design) Use ARC Demo time, science fairs, or scheduled time to check for design quality issues.


 Announce Verification time: (Eng) Turn on the features for MEDIUM_STAFF (or our team specifically) on Hatch and/or Prod. Announce in #arc and the project channel that the feature is ready to be verified and what the verification window will be (half day minimum). Product, Design, and Engineers will test the flows during the window and track bugs in the Project Management tool (Asana, Tracker, etc).

 Verify with design and product that the bugs are addressed, preferably in aggregate.

 If you don’t know if your project is Big or Small, ASK. If you think your project is small, but would still like a dedicated Bug Bash, feel free to do so.

You’re about to ramp something - woo00ot! But wait… how will people 

 about the awesome thing that you shipped?!


(Eng) Set up the ramp at go/experiments. 

 if you’ve never done this!! If this is an A/B test, check that you’ve entered the right exposure event with Product Science AND that you’ve checked the “Generate experiment results” box.


: (Eng) Once the change has gone to out to production and you have initiated the external ramp via go/experiments, announce the ramp in your team’s slack channel (e.g., #arc) and let fred visible-changes know so it also appears the 

: (Eng) Give additional details of the ramp in 

 so the product science team knows that the test is live, and can monitor the experiment pipeline.


: (Product) The PM should announce the ramp in #prod-announce, including a link to the product spec or other reference doc.

: (Product) The PM should also keep User Happiness in the loop if the ramp might impact critical member functions (e.g., cancellation flow, billing history page) and check in throughout the ramp in 

 to see if there are any concerning tickets.

The above steps should be done EVERY time we start a ramp or move to a different ramp!

Launch Verification and Announcement Process at Medium

Ramp Verification & Announcement Process

VERIFICATION

Verifying Big Projects

Sign up to see more