We started with quantitative analysis first to quickly rule out potential bugs or technical issues, and try to narrow down to some potential hypotheses to go deeper during the qualitative phase.

Reforge has a somewhat seasonal business based on when we offer live cohorts to our members. Typically, we’ll see key acquisition metrics (traffic, applications, paid members) spike in the few weeks before the cohort begins as members have increased urgency and willingness to pay. When benchmarking performance and gathering customer insights, we want to make sure we’re comparing apples to apples, so we’ll usually look at segments acquired within N days of a cohort beginning.

Knowing your audience is crucial and structuring your message to suit them is critical — especially in remote-first environments. Reforge is entirely remote, so the intended audience of this document was the immediate project team. Had this been meant for the Executive team or shared more broadly with folks outside the immediate project, I would spend more time context setting and leading with some of the key insights.

Displaying price late in the buying journey is an intentional choice. Because Reforge is more expensive, we spend most of the early customer journey educating about the value of Reforge and driving demand to apply. This has then set context for the higher price when they view it later in their decision-making process.

Dan Wolchonok

Ginny Mineo

This is a recap of an analysis I did with the Reforge growth team to better understand why users hadn’t paid for a membership plan.

In April 2022, we noticed that Reforge’s new member acquisition funnel conversion rates dropped below historical benchmarks. 

We conducted an extensive quantitative analysis

 to identify a root cause, ruled out multiple technical reasons (ex: marketing campaigns, product bugs, shifting user types), but didn’t have a conclusive reason for the drop.

To help get to the bottom of this problem, we sent automated 1:1 email outreach to our core audience asking them why they hadn’t bought Reforge yet, and compared performance to 

 to share with the immediate project team working on the investigation, and summaries of it were shared out with our Executive team and at a company All Hands.

Sent automated 1:1 email outreach to a random sample of 500 of our ADNP (accepted did not pay) target audience from April 2022. This segment included our core audience of product/marketing builders and leaders who hadn’t viewed the payment modal and those who work at larger companies

Received ~100 responses (20% response rate) and categorized into top themes.

Compared responses to categorized responses of previous ADNP outreach from March 2022 and Fall 2021 to see if there were changes in sentiment:

The top reasons for not paying hasn’t changed that much from Fall 2021, but there was movement in percent of responses mentioning top issues.

Price was the top issue mentioned in April 2022, up from the 4th most popular in Fall 2021.

see the price for the first time after being accepted

, and then would turn to their employer for reimbursement.

Note: Macro environment was only specifically mentioned 1X, but did seem to permeate the other price questions.

Thanks for reaching out. When I first made the application, I didn't realize there was a fee associated with it. When I got into the portal for the first time and got hit with the subscription screen, I was a bit surprised and needed to figure out if that was an investment I wanted to make.

Right now I'm trying to see if my employer will cover it. If not, I'll need to determine if it's something I want to budget into my personal finances.

Reimbursement continued to be a popular theme

many more people seemed caught up waiting to pay (30% vs. 8% of responses) 

after hearing from their company about budgets/approvals.

To be honest, I applied for Reforge before starting the negotiations with my current employer if I get the membership sponsored. Hence, I am currently in the phase of discussing/negotiating with my employer. After I have been successful (hopefully soon); I will be happy to take the next steps, learn and have an exchange with the community.

I originally planned to expense the membership. Like many tech company, our budgets are tight enough right now that I couldn't get this approved. I don't remember too much about the details, but I thought the membership was a little expensive, and mainly I didn't know have a very strong sense of how much I'd get out of it.

 (usually tied to price concerns, but not always). Compared to Fall 2021, more users cited a lack of information prior to converting making them question if the paid product was worth it.

I looked at the website (quickly), and it was not clear for me what's offered and the price. I don't have a lot of time to research, that's why I wasn't even able to evaluate it.

Thanks for reaching out about this. I’m happy to provide some feedback. When I initially received this email my first thought was “wtf is Reforge? You asked for brutal honesty. I honestly don’t remember applying for this. I looked through my emails and saw that I did indeed apply around April 22, but never actually opened any of the other emails Reforge sent me. They probably just got lost in the hundreds of emails I generally get and I just forgot all about it. So today I read some of them trying to remember or figure what Reforge is and got a basic idea, but definitely not enough to get me to pay two grand for the service. I honestly just don’t know what makes this service worth the fee.

Some were confused by some core Reforge experiences:

 — both by the steps required to actually sign up and what applying actually “means”.

I wasn't 100% aware of what Reforge is, and was a little bit confused about what I was signing up for. When I got an email that I was accepted, I was thrilled to see what's there for me... Just to realize that nothing is there for me unless I pay. And then I asked myself: Wait, what I was accepted for? Did I just get accepted to pay you money? :) I cannot pay you money since I don't have any clue will that money be well spent. I expected to see some content when I got accepted but didn't find.

Thanks for reaching out! I was already a Reforge member, but I got confused when trying to log in on a new computer and picked my Google account instead of my LinkedIn account. I fumbled my way into accidentally applying again.

The whole login flow has been difficult to navigate every time I’ve needed to use it. I’d be happy to give more specific feedback about my experience if the folks who work on the flow would like to hear it.

 — potential members weren’t sure what programs were available and how they will fit into their schedule, and it gave them pause to buy.

I was reviewing the offer but I could not tell if I was offered a spot in the community only or if I was able to join the summer cohort?"

I don’t know when the programs are happening. I’m taking time off work and want to make sure the specific programs I need will be offered within that timeline - and I haven’t found it easy to understand when the different courses happen.

~30% of responses were from applicants in the acceptance experiment.

Price was a top issue for both audiences, but the experimental audience had a smaller percentage of people waiting to pay and get reimbursed.

Waiting to pay (19% in experiment; 34% in standard)

Getting be reimbursed (12% in experiment; 33% in standard).

A similar percentage of the audiences were concerned about the membership value, though the experimental audience had higher levels of concern about relevancy of the content.

Many responses loved the personal outreach, and were very candid in their feedback. A handful bought as a result of the outreach.

These qualitative insights were the strongest signal we had

 around major changes to our new member acquisition funnel in April. Based on this feedback, I suspected two things were happening:

Macro challenges (inflation + tech stocks dropping + layoffs) wer putting downward pressure on individuals’ and companies’ ability to buy. Individuals were now more price sensitive, and at our price point, need to seek support from their companies. Companies were still trying to understand the macro impact, so they were slower to support reimbursement processes (it’s also unclear if they might also lower L&D budgets).

We’re targeting more adjacent audiences than our core fans in Fall 2021, but our acquisition playbook hasn’t changed all that much since then. Users were less familiar with Reforge overall, were unsure of the impact it will have on them, and don’t view $2,000 as a deal. We need a different approach to sell them, especially in the context of the macro changes.

There were still a lot of things in our immediate control

 based on this feedback to improve this funnel ASAP, such as: 

Leaning in harder on Summer Cohort promotions ASAP. Customers should have the information they need to know whether to buy Reforge right now.

Continue nudging March/April applicants who may be waiting for reimbursement to go through, and test sending acceptance/payment nudge emails from a real person’s name.

Continue gathering this type of feedback, so we can have an ongoing pulse of how feedback changes over the next few months.

Funnel analysis feedback summary at Reforge

Background

Methodology

Sign up to see more