Whenever metrics look bad, someone will ask “have we done research into why this is happening?”. This is a great moment to be the person who already has the feedback and real tangible stories from multiple customers. It’s a great way to help your manager / skip level manager make a push for a solution, or help deliver the message yourself of where opportunities exist to make things better.

The result here is that you may be asked to lead a team to help solve the issue or you can help to evaluate whether the current roadmap will address the underlying issues.

Talking to customers is always a good thing and having real examples of why people are happy / unhappy or why they aren’t taking key actions is usually the catalyst for change within companies.

We have found that we get a much higher response rate when: 

The email looks like a normal email you’d receive from someone’s personal account, not a marketing email

The email is written in a way that feels like personal outreach without marketing speak and feels human

The email asks for a quick reply (instead of linking to a survey) 

I’ve received higher response rates, more opinionated thoughts from users, and generally more actionable findings. I’m always surprised more people don’t do something like this.

Many non-technical people may be scared away by this step. The most productive team members will find a way to get this data. There are a bunch of ways to get this:

Pulling lists of users manually from your BI tool (any company should have a standard BI tool that enables this)

Pulling lists of users from tools like Amplitude that lets you specify criteria like (have done X but haven’t done Y)

Asking a data team to help populate a spreadsheet for you automatically using their existing tooling

Asking someone technical to pull data from a one-time list. This isn’t great as the data will get stale but you can always plead ignorance with users.

I can change my email template in the outreach to customize it. I might use very different language if they’ve seen the payment modal but haven’t purchased vs. they haven’t even seen the payment modal.

I can use this data to segment my analysis later. Price may be a reason someone hasn’t converted for smaller companies as they may not have budget, but navigating company bureaucracy may be an issue for people at larger companies. That segmentation may be helpful when thinking about strategies to tackle the user feedback.

You don’t have to use Zapier to do this, you could use whatever marketing email tool your company uses. The upside to this approach is that you can easily do it yourself if you know how, but if it’s more complicated than you would like than you can always use your company’s marketing tools.

The upside to using your company’s tooling is that you can leverage templates, ask for help from an expert, and factor in any exclusion lists that marketing is already using.

The downside to doing it yourself is that you may cross paths with someone else doing something similar, the only way to ensure that you don’t step on someone else’s toes if you coordinate it through a centralized tool. In the world of startups, speed and execution is better in my opinion as you’ll learn faster and be able to improve faster.

This type of analysis isn’t for a long term process, but for understanding why something is happening. Once you find that out, then you can build more sustainable processes and experiments with a broader set of tools and teams.

Depending on how you set this up, you can have a constant stream of emails to respond to coming to your inbox. It can be quite a lot of work if you email 100 people (going back and forth takes time) on a regular basis, but there’s nothing stopping you from emailing a small number of people on a regular basis to have a continual stream of customer feedback flowing directly to your inbox.

I very specifically chose the word brutal as I don’t see many subject lines with this language and I hope that it sets the tone for the type of feedback I’m looking for. Most people feel bad giving you negative feedback in situations like this and I always want to leave the door open for them to be honest and volunteer their true feelings. The last thing I want them to do is sugarcoat their feedback, and my hope is to give them the space to do that.

You can tell that the bar for people and companies soliciting this type of feedback is quite low - multiple people compliment the email asking whether it’s manually typed or automated, and it really helps to reinforce that you find their feedback helpful when you respond from your account and quickly.

This is the most important part of this analysis. You want to have a simple set of visuals that tell your story. In this example, it was “Why aren’t people buying access to Reforge?” and everyone wanted to see the main buckets the feedback fell into.

If you do this analysis more than once, the next thing to do is report on how this changes over time. Ideally, you can point to how initiatives have improved conversion rates.

Example: one of the things that people don’t do enough is ask their companies to pay for Reforge for them. Many companies have budget allocated for people to develop their skills, and often times it goes unused because people simply don’t ask. We’ve tried a lot of things over the years to empower people to understand whether their company has sponsored people in the past, what their company budget might be, and ideas on how to pitch the investment to a manager. We have built a lot of experiences through marketing and in-product that attempt to empower people and we’ve seen improvements over time in this area.

Dan Wolchonok

This feedback collection process for analyzing customer input helped me get the people's attention, re-prioritize roadmaps, and rally the team around how to allocate resources for work on a payments page.

, we analyzed our funnel and didn’t find anything broken in terms of the experience and our baseline metrics. 

The fundamental takeaway was that we saw a weakening in our conversion metrics

 across all segments and observed new behavior starting in April. The next logical step was to perform a qualitative analysis of feedback from users to understand what was driving the change in behavior. 

This document outlines the technical process we used to perform this qualitative analysis on the accepted applicants from April 2022

Our goal was to speak to a very targeted audience of people that demonstrated net new behavior (lower conversion rate from accepted applicant to paid customer). That audience was people who submitted an application in April 2022, were accepted, and hadn’t yet paid for Reforge by May 2022.

This is the outline of the process we followed:

Populated a list of accepted applicants from April 2022

Populated a list of accepted applicants that have converted to paid (so we don’t survey them)

Emailed back and forth with people about their feedback

Pulled the responses into a google spreadsheet

Aggregated that feedback per person so others can read all of the feedback in one place

Bucketed the feedback into categories and report on the trends using pivot tables/charts

We last ran this playbook in 2021 as we switched from a transactional model to a subscription model for Reforge. We wanted to get feedback about what was confusing to prospective customers as we switched from selling a single course experience to a SaaS all you can eat type offering.

Step 1: Dump contact information into a "CRM" spreadsheet

 to dump information from my data warehouse into a spreadsheet. It runs SQL that piped in these columns:

The date they submitted their application

Their function (product, eng, marketing, etc)

Their job title (ie: Senior Product Manager)

The number of employees that work for their company

The source of their application (our marketing, social media, one of our partners, etc)

The number of people from their company that are currently members

The number of people from their company that have ever taken a program with us

We sync'd another spreadsheet that listed all users who converted to the membership. We created a column that would look for every application that was created and ensured that users hadn't converted to our membership. This would ensure that we wouldn't email someone who converted to our membership.

The columns in this data set allow me to do many different kinds of surveys:

Is the feedback different based on company size? Bigger companies may have bigger reimbursement policies.

Is the feedback different based on their role or seniority? We may do a better job of describing the value of our programs to Product Managers than Software Engineers.

Is the feedback different based on how they heard about Reforge?

We can tailor the outreach based on whether they have viewed the payment modal or not.

We can tailor the outreach based on whether others at their company have participated with Reforge in the past.

 so that anyone who's sending outreach can quickly pick people from the list without affecting what others view when they open the spreadsheet.

We then create a workflow in Zapier to send out a customized email message based on the data that is most impactful for my survey. The first branch of the workflow splits on whether or not they've viewed our payment modal:

This is the message when someone hasn't seen the payment modal:

We won't go into the process of how we put email addresses in a spreadsheet to email them, as that's fairly straightforward. But we took people from the CRM sheet and put them in a separate sheet that triggers the Zap that sends the email.

One of the things that we love about this type of process is that the responses show up in the sender’s inbox and we can respond to them as we would any other email. The responses can be filtered to a label so they don't clog up your inbox but you can use whatever workflow you’re already using with your email client. It feels so personal and we hope that it comes across that way with these audiences.

One of the things that we hold ourselves (and our teams) accountable to is how close we are to the customer in terms of understanding what's important to them, how they see our product, and understanding if there are gaps that we can improve on.

Here's an example of an email thread that we found to be helpful as we consider the many different sub-specialties in any function like product management, marketing, design, or engineering.

 from someone who didn't see how the offering was specific to them:

As we email back and forth with people, we also using Zapier to pull the responses into a spreadsheet so we can share the results with others on the team. Here's the zap:

When an email with the particular subject line of the feedback emails comes in, we pipe it into a spreadsheet in the CRM google sheet. Here's an example of how we create a row in the spreadsheet with the details of the email:

This is actual feedback from a potential customer.

We then created a new worksheet that aggregated all of the feedback from each person. This was quite easy! In column A, we put in a formula that lists the unique email addresses for anyone who responded to the feedback survey:

Then in column B we wrote this formula to concat every email for each of the people in column A:

=ARRAYFORMULA(TEXTJOIN(char(10) & "--------- End of Message" & char(10), true, if(Emails=A2,Response,"")))

Don't worry about learning all of the details of the formula, other than it looks for emails from the same person and concatenates them together into a single cell. Here's an example of two messages combined into the same cell for the contact:

You can see that the person reacted positively to the message

, and responded after a follow-up question:

For each person (and row) there are multiple things that we do. We may end up @ mentioning our colleagues to follow up with individuals who had particular questions (sales teams, customer support, marketing) if that’s warranted, and we also spent time categorizing each person’s feedback into a bucket. With each person in a bucket, we can then do reporting on the high-level trends that we’re seeing. In the third column, we put categories that help us understand the reasons why they didn’t pay. In the example above you can see that this person is waiting to see if they can have it reimbursed by their company and want to pay. There were actually two emails exchanged with the person, the second one is a response to our team asking about their expense policy that is helpful for the team to understand.

Once we’ve put a category for each person, we then generated pivot tables to understand the trends of why people aren’t paying.

Once we have this data collected in pivot tables, we take that data and summarize it for stakeholders in a summary.

This process was on the more complicated end of the spectrum of our processes to get feedback. You can always just pull a CSV export of email addresses and send them a web form, but this was another opportunity to feel accessible to customers, respond to them in real-time, and the ease of sharing and reporting makes this a compelling process.

While this may seem fairly complicated to set up, once you have it set up it's very easy to tweak it to get the feedback you want and it is very flexible and easy to share with others.

Automated user feedback collection at Reforge

Background

Process

Sign up to see more