We had seen some concerning trends in our quant data regarding sign ups and we were trying to understand whether the friction was in the actual user experience or if there were deeper reasons folks weren’t converting their subscriptions. We were focusing on going through the experience of trying to pay and using that as a catalyst to see what came up there.

This was the first in a series of studies examining some concerning areas we were seeing in our data that were having substantial impacts to the business model - starting with acquisition and moving onto retention. Ultimately, the learnings we got from these studies resulted in a rethinking of our pricing and packaging. It was a domino effect as one study opened up new questions/hypotheses (and we received more quant data).

I put the learning objectives, which become our anchor for the whole process, up top. Everything trickles out from being as concise as possible in what we’re trying to learn from our study - from how we’ll set out to learn these things, to specific questions that we’ll want to ask. If I were to do this again, I'd probably even narrow them down more. My general rule of thumb has been to focus on learning 1-2 things per small scale study.

We’re spoiled when we want to do user research with our existing members, because we have great relationships and easy access to them via Slack. Churned users are the hardest to recruit, and these ADNP users were somewhere in the middle. This $50 gift card incentive is usually enough to get people to talk to us. But of course, incentives are a double edged sword - as the motivation to participate in the research has changed. So it’s important to spend time during synthesis to make sure you’re taking this into account (especially when considering how much you weigh individual feedback).

With most interviews we didn’t get through a majority of the real prototyping because pretty quickly the reasons for not completing payment poured out. This was ok because we were super anchored to the learning objectives (which weren’t inherently pure usability). Using the prototype as a catalyst to dig into their core motivations and frictions was a great way to ground the conversation. It’s a great example on how clear learning objectives can allow you, and the other folks working on a study, to be flexible as you get into the nitty gritty to still achieve the main result - learning something!

When coming up with the interview questions, I hate the “what would be helpful?” questions because those are too abstract. We want to know where in the process did you get discouraged? Where were you on the site? What types of things were you looking for? We want to tie the questions to something concrete.

A lot of the best information and spiciest insights actually come once the interview is over. Sometimes during the interview, people are trying too hard to be helpful and be a “good” interview subject, so they filter their responses. When the interview is over, I like to leave a lot of open space, maybe a long pause and an open-ended question inviting them to share anything else that comes to mind. They’re often more honest about their real thoughts and motivations once the pressure is off.

Sean Baeyens

Kevin Weiss

Adam Green

As Reforge changed their subscription model, I needed to learn what pushed some users to successfully complete payment, while others did not. This moderated user research project used Zoom interviews to learn just that.

 project in order to gather more evidence around the user experience (including pain points, friction, user sentiment, etc) for users who have accepted but have not completed payment (ADNP). We’d like to observe users starting from the accepted state and completing the payment process while narrating the experience live.

We believe that accepted users are missing key information about the value prop and/or how to pay for their membership (for instance, reimbursement) that prevents them from completing the payment process.

When do users feel ready to complete their payment? What gets in their way before they are ready?

Do users feel they have enough information about Reforge’s benefits and value when they reach the payment modal? Why or why not?

Do users feel they have enough information on how to pay for Reforge membership when they reach the payment modal? Why or why not?

How much do users know about their company reimbursement policies?

Do users know the price of Reforge membership before they reach the payment modal? If so, where did they learn this?

Participants are at least somewhat motivated to join Reforge since they completed the application process.

Maybe also talk to users who completed the payment process as comparison?

Users who did not complete the application or who were rejected

: 👋 Time to chat with the Reforge team?

I’m [Interviewer Name], a [title] for [company]. Our team is looking to gain insight into users' experience with Reforge and their decision-making process on whether or not to become a member. We want to make some improvements to this experience, and your input would be incredibly helpful. 

For the research, we’ll be running a 30-minute moderated usability test over Zoom. We’d like to ask participants to share their screen and go through a prototype while narrating their experience live. 

I’d love to connect with you sometime between the weekdays of April 18-29 if you have a free spot in your schedule. I know your time is valuable, so as a token for your participation, we’re offering 

Feel free to grab a time that is good for you here! 

[Your calend.ly or other scheduling link].

Breakdown into steps instead of one task flow

After the testing, ask them the rest of the interview questions or any other follow-up

Hello, my name’s [Name] and I’m a [Title] from [Company]. I’ll be leading our interview today.

[Warm up - ask about day, build rapport ~ 2 min]

Great - well first I’d like to start by thanking you for making the time to speak with us. Your feedback is super valuable and appreciated!

Before we get started, is it okay if I record our conversation for note-taking purposes? The recording will only be used to help us in our research, and it won’t be shared with anyone except those with a need-to-know.

As I mentioned over the email, I'm looking to learn about people's experience with Reforge and their decision-making about whether or not to become a member.

To start, I’ll be sending over a prototype over the zoom chat that we’ll be going over together.

Is it ok if I ask you to share you screen of the prototype so I can watch how you click through it?

The flow that we’re going over today is 

accepting your Reforge invite and completing the payment process

While we go through the prototype, if you could pause on each page and talk aloud some thoughts you have that’d be great. I’ll also be asking questions as you go through each page to dig a little deeper.

Do you have any questions for me before we begin?

Before we go over the prototype, I want to start off by asking you how you found out about Reforge.

Set the context: You applied to Reforge and got accepted, and this is the email you receive.

What information would be valuable to you here before you make the decision to join Reforge?

What did you expect to see when you accepted your invite to Reforge?

What information do you need to know about Reforge’s benefits and value to accept your membership?

When you see the price, what are the biggest thoughts, questions, and concerns you have running through your head?

Were you already aware of the price of Reforge membership before you got to this step? If so, where did you learn it?

How aware are you of your company’s reimbursement policies regarding your L&D budget?

How ready do you feel to complete your payment?

What’s getting in the way of you being ready to pay?

Can you talk me through the process of what it was like to talk to your manager to get approval?

Before going over the prototype, I want to start off by asking how you found out about Reforge. Why did you decide to join?

How did you find that information when you decided to join Reforge?

What are some steps you had to take to ensure that you could go through with the membership?

Payment Flow Usability Test at Reforge

ADNP Study

Summary

Hypothesis

Learning Objectives

Sign up to see more