You Cannot Be Data Driven Without Experimentation

People Overestimate Their Experimentation Skills

Many product managers, marketers, and engineers think they know how to run an experimentation program, but in most cases, they don't. What they know how to run are tests.

A test is an action that generates an output. Experimentation is a broad, repeatable system that happens to include testing as one of its steps.

As part of the Growth toolkit, Experimentation has long been associated with optimization — a tool to help you improve the edges of a product experience. But the most successful tech companies build Experimentation into the very heart of their culture, treating it as an integral part of the marketing and product development processes.

That's because Experimentation is one of the most valuable sources of product knowledge for a company. It's not there just to help you optimize landing pages, but also to help you resolve some of your biggest unknowns. It is one of the most powerful methods for informing the core strategy.

For those that deeply learn Experimentation, it can be a substantial career accelerator. With an understanding of how to build and implement an Experimentation system, you will become a center of excellence for your company. You'll find people coming to you for advice and perspective. An experimentation leader ultimately provides the knowledge that helps a company bridge the gap between perception and reality.

Fareed Mosavat: Experimentation is a tool of humility.

I've heard every straw man objection to experimentation — that it only works for small changes and optimizations. That there's no point running an A/B test if you know you are going to ship it. That this feature is obviously better than that one, so there's no point in taking on the extra work. Unfortunately, the result is that many product teams treat experimentation like QA and a way to cover the downside, instead of the strategic tool that it is. I've written previously about this.

Running a great experimentation process is an incredibly valuable Product Management skill, even for PMs who don't work in growth or marketing, because it helps you connect the dots between your intentions and reality.

Experiments remind us that even the best product thinkers in the world are surprised by their customers sometimes. They are a tool of humility, not decision-making.

Being great at building strong product experimentation loops can be a huge career accelerator for any PM, because it is hard to do well. Most people think they know experimentation, but they don't.

It starts breaking down problems into testable hypotheses. Running an A/B test without a hypothesis about the why behind your predicted result isn't strategic or smart. It's throwing spaghetti at the wall and seeing what sticks. Many teams skip the hypothesis.

It's about going beyond "should we ship it" to develop deeper insights about your users to determine "what should we do next"? That's strategy, not just tactics. But you have to focus on the overall "learning loop" that experimentation enables, not just the outputs. That enables smarter strategy, better decisions, and exceptional products for your users.

If you develop the skills and intuition to design great experiments to generate learning loops, you turn yourself and your team from just another feature team into a source of valuable customer insight and potential strategy for the whole company.

Elena Verna: Don't reward luck. Reward scientific thinking.

There are two reasons why Experimentation should be a core process for product and marketing teams.

First: We initially build products for our core users, but as companies scale the top-of-funnel starts to expand. We tend to know our core users well, but our new users often have different needs and desires. The gap between your perception of users' needs and the reality of what they actually want starts to widen. You can guess about these non-core users and rely on luck. Or you can use Experimentation, applying the scientific method to bridge the gap and deliver exactly what your customers need. Don't reward the culture of luck. Don't reward the culture of intuition. Reward the scientific method. When done correctly, experimentation delivers truth, not opinion.

Second: Everyone wants to be data-driven. We look at quantitative data (what's happening) and qualitative (why it's happening), and we want to use both as evidence for any decision we make. But those two forms of data still do not provide the full picture, because those data sets are not connected. I challenge you to think about Experimentation as a necessary third data set that truly makes your organization data-driven. It enables you to directly connect the What and the Why, providing you a full, robust picture of your business.

Matt Greenberg: Experimentation should be in everyone's toolkit.

Experimentation is a critical part of your toolkit. It doesn't matter who you are — business person, product manager, designer, engineer — you need to be able to make decisions based on hypotheses, data gathering, execution, and retrospection. Otherwise, you're flying blind. You need something like experimentation to drive the way that you systematize the work that you do.

It doesn't matter if you do it qualitatively — you talk to users, build hypotheses, execute, talk to users again, be retrospective, and repeat the loop. Or you do it quantitatively — you read the data, build hypotheses, execute, read the data, conduct a retrospective, and repeat the loop again.

That's what experimentation does. Without it, you're narrow. You build shitty stuff. You can throw anything out there and potentially get lucky, but you never know. There's no outcome metric to it.

Fundamentally, we're talking about filling out the toolkit people need to have to be successful leaders. Read every VC who writes about starting a startup; they say talk to users. Why? Because it's how you get data to build hypotheses and to experiment. You look at every successful large company; they have a huge experimentation team.

There's a common blind spot I see with people in tech companies. They make tons of decisions, they think of themselves as data-driven, but their decisions still end up being narrow. They lack the structure of a truly scientific method, and they leave either quantitative or qualitative data by the wayside. This leads to fragmented, incomplete thinking.

I see these two narrow profiles. I see people leave qualitative research to the side. They let user researches talk to all the users. They feel like they already know. Or, I see them disdain looking at data. They can't run their own queries on the data warehouse. They wait for an analyst to tell them what matters or what happened. They don't understand how to take both these inputs in for hypotheses and retrospective analysis. Both profiles leave wins on the table.

If you want to be dangerous in your career, you need experimentation in your toolkit.