Slow Down to Speed Up Your Product Analytics

The Breakdown

In our conversations, we discussed these three key steps:

Step 1: Map Out the End-to-End User Journey

Step 2: Collect Hypothesis Driven Data Collection

Step 3: Master the 3 Components of a Healthy Data Infrastructure:

• Data Quality
• Data Democratization and Self-Serve Capabilities
  • Data Dictionaries
  • Investing in people’s data education
• Exploratory Analysis through a Diverse Set of Lenses

Before we dive in, let’s quickly level set.

What is Product Analytics Again?

Product analytics is the process of analyzing user engagement and behavior data, or how users engage with your product, so that you can improve and optimize it.

In order to ensure you are building a sound data infrastructure to perform impactful analysis, you must begin by performing the three key steps that most product orgs miss.

Let’s dive into the first one.

Step 1: Map Out the End-to-End User Journey

Before you do anything else, Divya says to start your product analytics quest by mapping out the full user journey — understand the “now” of your product.

Patrick says having fragmented end-to-end journey data is one of the biggest mistakes product teams make.

“90% of the teams I talk with don't have full visibility into their end-to-end customer experience as they have separate marketing and product analytics solutions, and poor instrumentation. This creates an incomplete picture of their business resulting in siloed decision-making.”



— Patrick Thompson, Director of Product at Amplitude

Without a clear picture teams often focus on the wrong problem to solve or develop inaccurate hypotheses, only to learn that they don't have the whole story later on — this results in wasted effort and missed opportunities.

In other words, those user journey gaps become blind spots.

At this point, the question you might be asking yourself is why are directionally accurate hypotheses so important? This brings us to our next key ingredient for successful product analytics — hypothesis-driven data.

If you don’t collect hypothesis-driven data, you run the risk of gathering useless data that doesn’t unlock the insights you need to help you make product-enhancing decisions.

Step 2: Collect Hypothesis-Driven Data

Collecting hypothesis-driven data may sound simple, but so many orgs today still miss this step completely.

Orgs should pinpoint solid hypotheses around user behavior, collect the data around those hypotheses, then use product analytics and experimentation to prove or disprove those hypotheses.

The data we collect is critical because it’s what we base our analytics and our experiments on. We use the analytics to show us correlation, and experimentation to help us prove causation.

But what usually happens instead of a hypothesis-driven approach is a mad dash to collect as much data as possible without a focused goal, leading to what we call service center mentality where product teams misuse their data teams’ skill sets and expertise.

Service Center Mentality

Endless data queries are the unspoken culprit of meaningless analytics.

And yet, most data teams find themselves smack dab in the middle of a service center nightmare, responding to endless data requests from different pods across the product team that don’t seem to have any rhyme or reason.

Divya says the teams that succeed are the ones that are hypothesis-driven, not the ones who perform data theater and collect data for data sake.

Like we mentioned, hypotheses should be the driving force behind a data infrastructure that’s harnessed around answering specific, strategic questions. Once those answers are found, they can then be used to make product decisions focused on solving core user problems.

“There’s nothing worse than having your data teams engage in an endless cycle of random data queries. Teams waste valuable time performing useless analysis that doesn’t actually move the needle. It just creates a very negative self-serving loop. When you can connect the data you're collecting to prove or disprove a hypothesis, that’s when it creates the speed and urgency for analysis that defines a strong product analytics organization.”



— Divya Chittoor, Product Advisor & Former VP Product at Lob

Ben says a well-informed hypothesis will include the word “because.” That “because” describes the “why” behind the hypothesis, and it's something that you should be able to evidence in prior data — quantitative or qualitative.

When you can validate your hypothesis with data and experimentation, you're really uncovering the true story behind “why” a user is engaging with the product in the manner that they are, and that’s when you can make impactful, data-informed product decisions.

Nurture an Analytics Mindset in Product Managers, Not Just Data Teams

So how do we get our orgs to embrace a hypothesis-driven approach

Elena Luneva, Chief Product Officer and General Manager at Braintrust, believes a strong hypothesis-driven approach to data collection begins with nurturing an analytics mindset for the product management team and embedding that mindset into the culture.

Instill in your team the need to start every data query conversation with the why and to involve your data team as early as possible.

Furthermore, define what product analytics metrics you are hoping to move and define how the data will prove or disprove the hypothesis. By doing this, you set the foundation for quality instrumentation.

Once you have those hypotheses and questions at the forefront of your data queries, you can more strategically and confidently select the data you’ll require to perform impactful analysis, the tools you will need to track that data, and ultimately be able to implement a defined data tracking infrastructure.

Product Analytics and Company Alignment: What You Have to Know to Master Your Product

Ben adds that it’s not only important for individual teams to be aligned on the why, but for the entire company to be aligned on the two things you must know to master your product:

1. What are the most important problems to solve for, and
2. What is the most important data to pay attention to in order to get the information and understanding we need to solve those problems

When the company isn’t aligned on the why, different pods or teams begin requesting data for their own agenda, instead of data that matters to company-wide goals. Without identifying KRs and contextualizing what you are looking for, dashboards begin to lack focus.

Patrick says it’s easy to fall into the trap of tracking all things, but slowing down to define your business questions, Northstar, constellation metrics, and events allows you to prioritize the user behavior you track based on those essentials.

Once you’ve clearly defined your hypothesis, you can start to build an infrastructure composed of three key principles — data quality, data democratization, and diverse exploratory analysis.

Step 3: Master the 3 Components of a Healthy Data Infrastructure

Now that you’ve mapped out the full user journey and nurtured a hypothesis-driven approach to gathering data, we can now move on to gathering and using the right data to perform your analysis by setting up a healthy data infrastructure that consists of three components — data quality, data democratization, and diverse exploratory analysis.

Data Quality

Part of the equation to a healthy data infrastructure is ensuring high data quality — or, collecting clean data.

“Data quality is the achilles heel of product analytics. It comes up in pretty much every customer conversation at Amplitude.”



— Patrick Thompson, Director of Product at Amplitude

Both Divya and Patrick used the same adage: garbage in, garbage out.

If your team isn’t analyzing clean data — whether because the data isn’t accurate in some way, is riddled with test data, or you’re simply tracking the wrong thing — you’re just slowing down your product’s value creation. Do not rush this step.

You need to ensure there aren’t any breaks in data streams.

Three common data quality challenges:

• Messy data: Inconsistent naming conventions and human error that happens during instrumentation that leaves PMs and analysts with messy, if not useless data.
• Analytics bugs: When tracking goes untested before launch, leaving analytics code vulnerable to bugs or data loss as the product evolves.
• Lack of ownership: When there isn’t a clear sense of who owns the data collection and analysis, things slip through the cracks and you risk unveiling inaccurate insights.

Once you ensure high data quality, data democratization is the next key component to developing a healthy infrastructure. It’s critical to put that clean data in capable, data-literate hands to ensure faster, impactful product analytics — and those capable hands shouldn’t only belong to your data teams.

Data Democratization and Self-Serve Capabilities

High-quality tracking isn’t enough. Product people need to know what the data means in order to interpret it or make actionable insights.

What is data democratization?

Data democratization is when an organization makes data usable for all employees and stakeholders — not just data teams — and emphasizes the need to teach them how to work with data, regardless of their technical background.

Patrick and Divya say that most companies default to relying on data teams to translate gathered data to product teams. And yet, leaving data to the technically versed is a sure way to slow down impactful product decision-making.

Instead,

1. Make data literacy a part of your product team’s career guide, and
2. Publicly celebrate when folks make data-informed decisions.

Self-Serve Capability Speeds Up the Decision-Making Process

Divya says it’s important to prioritize setting up your data in a way that teams can read it themselves and easily find their own answers in the data.

This accomplishes three things that ensure high-quality impact, faster:

• Empowers product teams to self-serve and answer their own questions, speeding up the decision-making process.
• Liberates data teams from the chains of being a service center so that they can focus on delivering deeper level analysis for the really complex data questions.
• Allows data teams to also focus on maintaining a clean and orderly data infrastructure.

When data dashboards are built to self-serve, you’re more likely to get diverse perspectives on solutions to the problems you're trying to solve.

Data teams should be working alongside product teams, embedded into the problem-solving, and up-leveling the underlying quality of the product team’s analysis. Patrick says to think of a data team as someone you can lean on to do a peer review of your analysis, checking the quality of the insight.

How do you set up self-service capabilities?

Data Dictionary & Style Guides

Standardizing a data dictionary is a great way to enable data literacy and self-serve capabilities within your infrastructure.

“I would say the most important thing is having an accurate and up to date data dictionary of tables and columns. It's amazing how self-serve things can be when you have that well defined dictionary and people know where to look. At Facebook, the data dictionary was well-defined. It let me and my team do a lot of our own hypothesis testing and freed up engineers and data scientists to do the more complicated things.”



— Divya Chittoor, Product Advisor, Former VP Product at Lob

The data dictionary is one of those things that is different or unique across every single company, so it’s important to label and define things as specifically as possible within your organization so that everyone in the company is speaking the same language.

Patrick says if you don’t slow down and take the time to standardize and codify your data dictionary, you put your analytics journey at risk.

It’s important to provide context around your analysis, like dashboards and notebooks, or you leave them vulnerable to interpretation without guidance on how the data should be used internally.

“People see what they want to see and often use data as a blunt force weapon. Your data dictionary is a solid tool for mitigating that risk.”



— Patrick Thompson, Director of Product at Amplitude

Elena also re-emphasizes the importance of using the same words to discuss data and ensuring they are spread across departments to help the company focus on the right things, preventing people from chasing things that don’t matter.

Should your company invest in product analytics training?

There is a resounding yes to this question amongst our experts.

While data teams work to accomplish the painstaking feat of gathering and assembling clean, understandable data, product managers need to invest in their product people and train them in data literacy and product analytics.

“Data tools are worthless without investing in your people. Build the muscle first, train and invest in improving data literacy. Better decision making comes swiftly after that.”



— Patrick Thompson, Director of Product at Amplitude

In other words, create paved roads for product teams to adopt analytics by first providing templated data dictionaries, instrumentation style guides, and tooling processes; and then supporting that with education.

Snyk Example of Product Analytics Training

Ben paints that picture of what this can look like in the real world.

While at Snyk, he sponsored one of the growth teams to do an educational tour across core product and engineering teams, with content and examples tailored to each team and their domain. There were 101s and 201s on data-informed development and experimentation. Establishing that common ground and then providing these workshops helped develop the culture and ability around self-serve analytics and data-informed decision-making.

Beyond the education, the growth teams at Snyk created a paved road for behavioral analytics and experimentation that allowed other teams in core product areas to very quickly on-ramp with guidance, support, simplified SDKs, style guides, and more.

Barriers to Self-Serving Capabilities

Divya and Elena say simply getting started is a strong barrier to reaching a self-serve infrastructure.

Start-ups especially get stuck in no action simply due to the size and cost of the opportunity to be more data informed.

Divya adds that most organizations underestimate the time it takes to build that self-serve infrastructure properly and instead rush into quick fix solutions to their data problems.

To make sure you don’t make this mistake, it’s important to internalize a simple fact: Gathering accurate, clean data and then ensuring it’s understandable for product teams to truly make use of it isn’t easy.

“Driving insights to make sure you’re building the right product is what really matters. To do that, you need to make sure that people can go in and answer their own questions when needed. Focus on making it easy for them to read and make decisions. If you don't have analytics as part of the responsibilities of the core team, then what you end up having to do is going to the data team and saying, I have a question on how users are potentially interacting with this thing. You have to put in a request, they have to prioritize it, they have to understand your request, and then they have to get back to you in like a week, two weeks, three weeks later. It becomes a longer than necessary process when you don’t invest the time upfront to develop a self-serve infrastructure.”



— Divya Chittoor, Product Advisor, Former VP Product at Lob

She recalls a time when a team utilized a product analytics tool, Retool, and PMs could see how customers were going into the product and see how much time they were spending at different points in the user journey.

Expert Advice for Product Analytics Success

In the quest to become data-driven, many organizations make the mistake of prioritizing quick and easy data collection over building a healthy data infrastructure.

It’s clear this approach often leads to the collection of dirty data and a lack of data literacy within the organization. The result is a slow and ineffective product analytics process that fails to deliver meaningful insights and solutions.

To avoid these common pitfalls, it is crucial to invest the time and effort upfront to develop a solid data infrastructure. This means taking a step back to identify core problems and intentionally formulating questions that you want the data to answer. Simultaneously, it is essential to upskill your product teams and empower them to become data literate, enabling them to effectively analyze and utilize the collected data.

By following our three step approach to product analytics that we outlined in this article, your product analytics organization can transform into a well-oiled machine that consistently delivers impactful analysis and solutions.

Remember, while it may seem tempting to take shortcuts and rush through the data collection process, the long-term benefits of a healthy data infrastructure far outweigh the initial investment. So, slow down now to speed up later, and unlock the true power of data-driven decision-making for your organization.

Sign Up Now