Casey Winters On Pinterest's Retention Wins & Why 90% of SEO Advice Is Wrong

Pinterest isn't Facebook, and that's ok

A lot has been written about the One Metric That Matters, but businesses, and the investors that are watching them, still regularly fall prey to vanity metrics. Rather than mapping metrics to the core mission of the organization, many observers and even solid growth teams are still comparing apples to oranges.

For example, Pinterest’s 150 million MAU might not look like a lot compared to higher-frequency products like Facebook or Snap. But Pinterest is a very different beast -- one that CEO Ben Silbermann says is “more of a personal tool than a social one.”

Although it has social roots, the tool is fundamentally about a person’s relationship to content and ideas that they discover either on Pinterest or elsewhere. Logically, user acquisition doesn't grow virally in a way that a social network does. Instead of connecting to others, people are on Pinterest to connect with things to do and buy.

“What was so exciting to me about joining Pinterest from a business perspective is that it's a rare business where the revenue model doesn't conflict with the product engagement. People are there looking for ideas on things to check out and buy, and it turns out brands have really good ideas that they can show them that show them that are just as compelling as another, non-brand idea.”

Pinterest is a high-intent product, says Winters, and the advertiser tie-in is natural and even additive, not disruptive.

From a metrics perspective, whether for Pinterest or for any other company, growth leaders should look to the business’ core mission for guidance. Pinterest says its mission is to help people discover things they're interested in and then to do them in real life.

“What we had to think about was how do we know if someone found something they're interested in on Pinterest? The key metric that signals that moment is if they save it. As a result, Pinterest has always looked at saves as a key metric.”

The “Save” metric at Pinterest also provides valuable signal on growth and on the quality of the content on the platform -- is this content something good, and will other users also enjoy it? More Saves means it’s safe to surface a given piece of content to more users, and helps the company to understand and measure value delivered not only to an individual user, but to the broader network of users at the same time.

The North Star Metric for any business: you're optimizing for revenue or mission

But what if you're not Pinterest, and you’re tasked with determining what metric or metrics you should focus on? Pick the wrong north star, and it could become a costly misguidance for your business.

Winters makes a key distinction between mission-oriented companies and business-oriented companies.

“If you're mission-oriented like Pinterest, you try and find the metrics that most closely align to giving you signal that you're fulfilling that mission. On the other hand, you’re more of a business-oriented company trying to make money, then you try and find metrics that most closely align to you making long term revenue.”

In a previous role leading growth at GrubHub, Winters looked at how much incremental order revenue that service drove for restaurants. On the diner side of their marketplace, the key metric emphasized retention and measured frequency of repeat orders.

But, just as Pinterest itself doesn’t fit Facebook’s mold for the metrics that matter, every other business also has its own key areas of focus -- wedding sites and real estate sites aren’t strong on long-term retention, simply because the real-life events that they relate to are typically so infrequent. Determining the right retention for a business where you want to grow repeat usage boils down to measuring that action in the natural time period in which you think they should do it.

But, even though some businesses don’t lend themselves to retention -- you're not going to buy furniture everyday the way you might buy lunch everyday -- Winters says that it’s possible for great products to influence the natural frequency of usage as the product and its relationship with its users mature.

“Early in a startup, I target whatever the normal frequency is for that behavior the way people currently solve that problem. Long-term, I would see if it's possible to increase it and change the frequency metric accordingly.”

Winters says that Pinterest’s early metrics were monthly metrics, but the company has since shifted focus to weekly metrics as people’s behavior has become more frequent. The metrics that matter can and will shift as the stage of the company itself continues to develop.

Your growth experimentation process is going to change

Just as metrics grow with the business, so does a growth team’s process of experimentation. Early stage companies will have limited data and may not have an experiment framework that allows them to run great tests -- for example, you might want to show an experience to a certain subset of users and not another.

Instead, growth at the early stages means making big changes and measuring results pre-change and post-change to see what, if any, impact those changes created. If it’s immeasurable, then it’s likely that the change didn’t have the type of impact an early stage product needs to see.

As a company scales, growth teams start to test more incremental changes that can still make a major impact on the business. Experimenters can now isolate variables in each experiment and measure impact at an increasingly granular level.

At that point, says Winters, you need to have an A/B testing tool to measure the impact, preferably something built in-house, or default to an external tool if you're just getting started.

But many teams get stuck in a rut that they can’t test out of. The problem, says Winters, lies in the experiment’s design.

“One mistake I see some teams make at this stage is that they only do experiments isolating one variable at a time. Sometimes, you still need to do more divergent experiments that test a totally new experience otherwise you can end up stuck in a local maxima.

Or, you can slow down your pace of innovation even as your experiment velocity improves because instead of doing one experiment to get to your goal, you end up having to do 150 experiments to isolate every little change along the way.”

How Pinterest leap-frogged local maxima

The biggest wins in growth are often the results of a lot of little experiments, some of which may yield positive results in direct contradiction to industry “best practices.”

Every one of Pinterest’s major wins has arrived through experimentation, says Winters. In one example, the team built their own SEO experiment framework to help them figure out which site changes were resulting in search traffic bumps. The learnings there had a dramatic impact on traffic -- and that was just from facilitating the process of experimentation that was already happening.

Later, when the team saw that these new visitors were not converting into sign ups as much as they would have liked, they ran another test.

“We did a quick experiment where we stopped the user from scrolling after a while (the default action of a Pinterest grid is people scrolling). Instead, we pulled up a little banner that said, ‘Sign up to see more of this content.’ That experiment alone improved our conversion rate fifty percent, and with it every graph in the company kinked upward. It was pretty small to do, didn't take very long time, then all of a sudden you're on a totally different trajectory.”

Failed experiments in email and phone activation

It’s inspiring to hear stories about a fifty percent conversion rate bump from adding a signup call to action, but ultimately if you’re designing your experiments to only find wins -- rather than to answer questions -- then you’ll find disappointment, not growth.

But, says Winters, the learning from why an experiment fails is infinitely useful in helping you to find the bigger winners later on.

“I’m regularly surprised by experiments, and you’ll frequently hear me in meetings exclaiming, ‘That’s why we do the experiment!’ because something totally different happened than what we expected.”

These failed experiments mostly point out things that make sense in hindsight, says Winters. The team spent time designing an elaborate new email flow to help onboard new users with how-to content and getting them to download the Pinterest mobile app, which they noted as a corollary to higher user activation and engagement rates.

“Not only did those emails not improve our activation rates, but the email about the app in particular had more unsubscribes than app downloads. It was working against us pretty aggressively. That experiment helped us figure out that sending emails with relevant content -- matching why people joined Pinterest in the first place -- is what mattered to increasing activation, not teaching users more about how the service worked.”

Winters recalls another surprising experiment result when the team rolled out a change allowing users to use their phone numbers as a way to sign up, without email.

He feared that not capturing a user’s email -- and opting for phone only -- would lower their activation rate because they would lose out on a valuable channel to bring users back to the site, without substituting an alternate communication channel (like SMS) to make up for it.

“Those users activated at exactly the same rate as people who gave their email addresses, but adding that sign up option got us a lot more users. So it ended up just being a totally positive surprise for us.”

Retention beyond growth

Retention is by far the most important success factor for business. The challenge from a growth perspective, however, is that the responsibility to take care of retention can be shared between multiple teams. Growth teams may not have as much freedom to experiment outside of emails and notifications, which are a limited toolset for improving your retention, says Winters.

“Sometimes it's just about making the product better. At GrubHub we had gotten people to start ordering food online, and they were doing it regularly, but we measured through survey data that many of our users still didn't do all of their delivery online with us.

We asked ourselves if it was a product problem or something we could influence through product changes. It turned out the main reason that people were ordering via those restaurants without Grubhub was because they had been ordering from those restaurants prior to using GrubHub, so they maintained their old habits.

We thought if we could provide a small incentive to move those orders onto GrubHub, we could really change our frequency. So we created a game that you played after every third order, and with that game you had a one in four chance of winning free food. We launched it hoping it would break even as like a three month test, but the cohorts immediately started to bend upward and order frequency shot up and sustained. People just loved playing the game.

Even though we were giving people a lot of free food it ended up being profitable for Grubhub to do because of the increased frequency.”

At Pinterest, Winters saw more success with product changes that became growth wins. When the service launched outside of the U.S., the “Pin It” button was deliberately not localized because the company viewed “Pin It” as part of their brand.

But, one day someone on the team decided it would be worthwhile to run a test: what would happen if they changed “Pin It” to the localized word for “Save” for the different non-U.S. locales where they’d just launched?

“Activation rates shot up instantly. It even helped activation rates in the U.S. to change that button from Pin It to Save. We ended up using that as a signal to do a lot more experiments to make the product simpler and easier to understand for new users.”

Although those experiments were aimed at activation, says Winters, the learnings they yielded had an equally positive impact on retention downstream.

SEO as a superpower

SEO is something that's not entirely well understood even in the growth community. It's also an area full of experts selling silver bullets, says Winters. It’s this type of quick fix or “silver bullet” content, he says, that populates the internet when you try to learn about SEO.

“What's emerged is a group of people that are studying every detail of Google's algorithm and patents and inferring what they think google wants, and end up recommending very non-scaleable platitudes like ‘write great content, get links.’

As growth people, we do care what Google wants, but we also care about what their current algorithm rewards. Those aren’t always the same thing. We’ve tried a lot of best practices that you read about on the internet that actually drive Pinterest less traffic.”

Specifically, Winters says that he has never written any content for the purposes of SEO, for any of the businesses that he has tried to grow via that channel. Instead of reading internet best practices, Winters has a different take on the right way to learn about SEO.

“It's basically running experiments on the Google algorithm and seeing what they care about for your business.”

The Pinterest team ran SEO experiments with surprising results that taught them to question every SEO best practice. The team’s goal was to show not just Google’s algorithm that their content was relevant and authoritative on many topics, but users too.

Since Pinterest is a catalog of ideas in the form of images, they knew they needed text around their images to tell Google what concepts the images were relevant for.

Winters gives a counterexample -- if you were growing Quora, a site with an abundance of text, you don’t need even more text. A strategy of adding more text won’t have the desired impact on search rankings.

“Companies should ask, what are the strengths and weaknesses in my business, what is the algorithm awarding and not rewarding, and then a creative exercise to try and build product that will work for users as well as search engines that helps you get the most you can out of a channel.”

Winters is adamant; checklists don’t work. Instead, he wants to see his growth team talk to customers, brainstorm solutions, test those solutions, and then think analytically and creatively about results whether positive or negative. “It’s like treating the Google algorithm as another customer. Run on experiment on ‘them’ and see what it responds to most.”

From "Growth Hacks" to growth as a function

When Winters first started working on growth, at Apartments.com and early on at GrubHub, he viewed growth as many in the industry still do -- a collection of the smartest tools and tactics that, once understood, a practitioner could bundle together into a strategy that would maximize growth.

“It was a little bit like a Pokemon exercise -- gotta catch them all. Gotta learn about SEO. Gotta learn about email. All these different things. I now focus much more on understanding the current business I'm working on, where it's working, where it's not, and seeing if these tactics that I've learned can help, but if not, just invent new ones.”

And by “inventing new ones” Winters means changes as wide-reaching as building a totally different product, or fixing operational issues within the company.

“I no longer think that growth is a menu you pick from. I now think of it as a function.”

The role of that function, says Winters, is to connect the most amount of people to the current value of a product. He says rather than merely bringing more people to the product, companies should focus more on fixing the friction deep inside the product that’s preventing people from connecting to its core value. That effort may not have anything to do with SEO, email or any one popular tactic or channel.

Growth doesn’t fit into its own little room separate from operations, from customer service, from product. Instead, the growth team of the future is going to look a lot like one unified product team.

“When I arrived at Pinterest I would think, growth owns acquisition, activation, retention, resurrection, virality. I think in some ways it's healthy to start there. I think about how you start a growth organization, you usually start by those problems, like, "Oh, we need more users."

Or, "We need to improve our retention." Then you evolve into thinking about a more holistic funnel as you grow the team.

But, the next step beyond that is thinking about, “What are the current bottlenecks that are preventing us from being a more successful business and how do I apply an experimentation framework to diminish those bottlenecks?’ That’s where you know your growth team has matured.”

Special thanks to Casey Winters for his participation in this interview. To stay in touch, you can read Casey’s blog, or follow him on Twitter.