At Duolingo, we have two venues for product reviews: in-person and Slack. If the features are small enough, we'll just Slack it out: "Hey, here are the changes we're going to make." To keep things efficient even when we're dealing with bigger decisions, our in-person meetings are always time-boxed to 15 minutes, which is why we start with the meeting goals to make sure that we're all on the same page about what we need to discuss and get approved. Being really rigorous at the very beginning—I try to keep my overview to 30 seconds—helps structure the next 10 minutes and let's me make sure that we're focusing on the things people actually care about.

Duo is such an important brand asset that shifting our messaging to a different character felt like a real strategic risk; I honestly wasn't sure that it was a good idea at the time. A lot of our messaging is really positive and motivating—learning a language is hard, after all—and Lily, as a character, is kind of more playfully negative. Even though we were getting a lot of great feedback around her on social media, we knew we needed to decide on the right balance here.

At the time of this project, the retention team owned five or six notification channels, and one of our OKRs was to run a certain number of experiments around notifications every quarter. We had the infrastructure in place to make it easy for us to run these experiments. If they worked, great, we knew we could launch those templates. If they didn't, we could quickly move on and test new copy.

The level of detail when you're doing this analysis should generally be in line with the level of effort the project is going to require overall. Since in this case the effort was only me writing copy, it was enough to provide some quick, back-of-the-envelope numbers.

This project turned out to be a huge win for us in terms of ROI. As a PM looking for metrics to optimize, my first instinct was to keep hammering Lily notifications all the time. But we ended up deciding to test some other characters instead, and that helped us learn that it was the specific voice of Lily (along with one other character) that was really driving success for notifications. We wouldn't have learned that if we didn't have a culture where we're so committed to quickly testing as much as possible.

Maggie Glascott

Michaela Kron

Mary Van Ogtrop

Jackson Shuttleworth

We use lightweight specs and 15-minute product review meetings to help us quickly launch high ROI experiments like this one.

Spec: D8 Practice Reminder Notifications from Lily

Add 3 D8 Practice Reminder notification templates that come from Lily, rather than Duo

Align on adding characters to more notifications

Are there channels we shouldn’t add characters to?

How many notifications should come from our characters vs. Duo?

We want to add 3 new templates to the D8 notification (the notification which sends after 8 days of inactivity); instead of coming from Duolingo / Duo, we’ll have these come from Lily, in Lily’s voice, with her face as the notification image.

We’ve seen more and more users responding to our world characters, particularly Lily and Zari. 

We’ve done enough testing on our D8 notification to know that copy improvements to this notification can drive both engagement with the notification and sustained D7 retention post-send

, and we believe that the combination of Lily’s tone of voice and having the notification appear to come from a person/character will make it more likely that users notice and engage with the notification in the short term, and complete a lesson and sustain their streak in the medium to long term.

If a user doesn’t respond to the D8 notification, we turn notifications off, so making this as high visibility as possible (without being spammy) is critical for not losing users.

We’ll also be testing normal practice reminder notifications from Zari and Lily (all copy, both prioritized and deprioritized, can be found here [link redacted]). Otherwise, this will be our first attempt at out-of-app messaging with other voices

Previous iterations on the current D8 Practice Reminder reminder template have yielded positive impact on DAUs, CTR

We use one template for the D8 Practice Reminder notification, from Duolingo

Test adding three different notification templates from Lily; we’ll test each as a separate arm to see the relative impact

Note: Notifications will choose from a random emoji that all D8 Practice Reminder notifications are eligible for

D8 Practice Reminder Notifications from Lily

All users that get the D8 Practice Reminder notification

Make sure to analyze for users seeing the notification only

Lily’s voice will be more noticeable than Duo’s

The fact that the notification is coming from a person vs. Duo or Duolingo will make this notification more noticeable

Feature Impact: +1% improvement in Notification Sent > User Active on Duolingo funnel

Note: Because this will bring back more low-retaining users, this will hurt Current User Retention Rate; this is due to the mixshift of users who will now make up the Current User base, rather than us hurting the current user experience

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Everything below this line does not need to be discussed in PR and can be filled out after the spec has been approved.

List any new tracking events and properties that will be implemented. Make sure to use existing naming conventions and specify what triggers an event.

Product Spec for Notifications Experiment at Duolingo

Spec: D8 Practice Reminder Notifications from Lily

Meeting Goals

Background and Summary of Changes

Sign up to see more