On the old onboarding screen, everything required to launch a campaign is represented. But not everything that is strongly recommended is there. For example, to be successful, a customer needs to define their conversion segment (basically, what event do you consider a successful conversion?). That step was not in the old onboarding flow because it came later and was a bit complex. But we noticed that the people who didn’t do it were likelier to churn because they had lower performance.

We debated whether to include this step in the new onboarding flow. We ended up with a compromise: it’s included, but it’s optional. We also added a lot of helper content to this step. In the end, the percentage of customers who identified a conversion segment went up, so it ended up being a really good thing that we included it.

We used equal amounts of quantitative and qualitative data to support this hypothesis. For quantitative, we compared our signup-to-launch rates with industry benchmarks and believed we could do better. We also saw that people weren’t completing all the onboarding steps on the static page.

For qualitative, I’ll be honest: we looked at a lot of onboarding flows for companies we admired, and most of them had this type of conversational flow. Most growth teams will say they do everything objectively, but in reality, everyone looks at what everyone else does and tries to emulate it, thinking that someone has figured it out. Why redo all the research that a lot of smart people have already done?

At some point during onboarding, you need to identify that someone is a customer. That’s typically a somewhat arbitrary decision that gets made before you as a growth leader ever join the company. When we had the single-page onboarding flow, someone didn't become a customer until they completed all three steps.

But we began to run into challenges when we broke up all the steps into the conversational flow. We wanted to be able to switch pages in or out quickly and experiment based on who that customer was. But if the person going through the flow wasn't considered a customer yet, we couldn't retarget them! So we had to change the definition of “customer” to be earlier, which meant we had to involve the data science and data engineering teams. It became a much bigger issue than we thought, so I recommend thinking about this beforehand.

There's still a happy ending to this story, though, because when I moved to head up the growth team in marketing, one of the first things we did was to roll out a lifecycle campaign that was email and in-app centric.

From what I remember, we celebrated these results first, and then we kept rolling it out. We had to translate this to different types of customers as fast as we could. We saw that it worked, and there was no need to wait or run more A/B tests.

Julie Zhou

We ran this experiment at AdRoll to test the hypothesis that, rather than it being simple for a customer to see everything they needed to do for onboarding on one page, they actually found it overwhelming and would leave the onboarding experience. 

I managed the product team responsible for creating the self-service interface by which any customer who created campaigns on their own, without the assistance of an internal account manager, would experience AdRoll. The onboarding flow for a new self-service customer was obviously a very important funnel for us, since these customers wouldn't have the benefit of a manager to help them if they got stuck. The existing onboarding flow was this page that you see here — our belief was that it would be clear to the customer, there were no surprises, and honestly, it was just easier to build this way all on one page.

On the old onboarding screen, everything required

But we wanted to improve the signup-to-launch rate, or the rate of people who signed up for a new AdRoll account and then launched their first campaign. 

 was that rather than it being simple for a customer to see everything they needed to do on one page, they actually found it overwhelming and they would leave the onboarding experience. So to test that hypothesis, we built a new onboarding flow called “conversational onboarding,” since it proceeds like a conversation, and we ran an A/B test with 50-50 randomized traffic.

At some point during onboarding, you need

We saw an immediate 20% uplift in sign-up to launch rate, and ended up expanding the experience to 100% traffic a month earlier than we had planned since the winner was so obvious.

Conversational Onboarding Experiment at AdRoll

Conversational Onboarding Experiment

Old Onboarding:

New Onboarding:

Results:

Sign up to see more