We started thinking about MMM as we saw diminishing returns from incrementality and MTA. MMM is just another tool in the toolchest for marketers and this project was to explore whether it could be a tool that would be useful to us.

Those involved included our head of growth, director of user acquisition/demand gen, head of data, growth analytics lead, head of martech and a lot of stakeholder ICs across marketing and data.

I would say very few people understood these concepts (both MMM and HDYHAU) deeply until we wrote them down and immersed ourselves in them.

It was actually pretty important in my opinion to define a lot of these terms because they get thrown around in growth marketing somewhat carelessly. Never assume your audience has the same understanding of a problem.

Even something as nuanced as DF is an important thing to clarify with a cross functional team. You can't assume everyone around the table has a masters in stats. Even if people have taken stats and understand these concepts at a high level, they may no longer recall how they map to specific outcomes. In this case, something like this is educating people on how "the more variables you have, the less certain you can be about something you're testing" - a helpful lever to remind people of as you begin analysis.

This is a really helpful guidebook for people to determine if they even need MMM. Starting with the questions you're trying to answer is always the right move when you're on track to do something, but not sure why.

The reason this got lumped in with MMM is because HDYHAU was another form of attribution beyond what we had previously tested - we'd already done basic attribution, MTA and incrementality testing, so it represented the final frontier.

A way to think about this is that all of these methods for data collection and understanding are tools in the toolkit. They each are trying to help growth teams understand the sources of attribution and where to take future bets and allocate dollars.

There is no silver bullet to using one vs the other... it's like having a tool bag and deciding whether you're going to use the pliers or the hammer.

Two major things that I think are worth sharing out of this project:

Before and after this doc we talked to a TON of people about MMM to become experts. I wrote down a list of questions (included at the bottom of this document) and we went and talked to folks and asked questions for hours. This really helped us understand what we were doing and what we should/shouldn't care about.

We got really lost in "perfect delivery" land where perfection of the statistical model undid our ability to just ship something. In my opinion, if you have conviction around something like MMM and your business model isn't 100% a perfect match to the limitations of MMM, it doesn't mean you shouldn't do it, and it also doesn't mean you shouldn't ship an imperfect model. Building something quickly to learn and use as a lever or tool for continued progress is the name of the game.

Austin Hay

These two common frameworks are an important part of the marketing stack, so we decided to put this documentation together to get all stakeholders onto the same page as we implemented them.

RFC: MMM & HDYHAU at Financial Automation Platform

What are we trying to accomplish with this doc?

Our goal with this document is to summarize two key projects for measuring Growth efforts at the company. Both methodologies allow for a holistic view of our marketing efforts by being able to account for Online and Offline measurement (something lacking in MTA). With this being said, both projects have various requirements in terms of effort and output. We hope to align on the following:

 which project we should prioritize first

Write down the outcomes the Growth team is trying to achieve with both of these projects

Based on our objectives laid out in step 2, align across our teams (Data, Martech, and Growth) on how we will want to digest the outputs of these projects

 This is a great question that we should all ask ourselves. MMM, like Incrementality, is a tool for our holy alliance ( Growth, Data and Martech ) to make decisions about 

(1) how we make investment choices, and (2) the mix of those investments choices in marketing activities.

Before we talk about MMM further, lets rewind how we got here - and how most companies get here.

All companies go through a typical attribution evolution as they look to grow (ie spend dollars and attribute those dollars to their growth). It looks like this:

Basic Website attribution (URL, referrers) + page / event tracking

And within this stack, this is typically how teams evolve:

All in an effort to better understand how they spend dollars and how they 

 spend dollars. That is why we are here and talking about MMM.

With new, bigger and more sophisticated mediums for marketing approaching the comapny in 2023, it’s more important than ever to be aligned on this core thought: The purpose of S2S conversion, Incrementality, MMM and other advanced measurement methodologies is in service to [redacted], [redacted] and the Growth team. To help them understand how to spend dollars to maximize the value to the company (optimal growth). It is rare for marketing teams further along in their development to use one tool in isolation and oftentimes uses MTA, Incrementality, and MMM as a way to triangulate around the best answer.

Having positioned why we are here, let’s talk a bit about wtf MMM actually is:

Media Mix Modeling (MMM) predicts business outcomes through a statistical analysis using multivariate regressions, with marketing tactics and spend as variables.

Multivariate Regression f(x,y,z) = mx+my+mz + b

mz → Other variables; seasonality, economic data, inbound promotions, events etc.Micro vs Macro: Micro model would apply to a channel, macro model would apply to the overall business and all deployment of marketing capital.

Example: Company Growth wants to know how each paid media channel contributes to SQL pipeline, CW and CP. We collect data for each channel over a several month-year time frame.

We run a multivariate analysis on many points in time.

This attempts to create a model or function that predicts the output (SQL, CW, CP) when there is a change made to media spend in a specific channel.

Typically multivariate analysis only shows correlation, not causation, because they are based on historical data.

As a business becomes more stable, and predictable, however, this can become more reliable.

: The degrees of freedom in a multivariate regression equals n-k-1, where k is the number of variables. The more variables you add, the more you erode your ability to test the model (e.g. your statistical power goes down).

Cannot be used to inform tactical decision making at the sub-channel level.

By comparison incrementality is another way to measure marketing mix choices

Both MMM and Incrementality measurement require a certain level of volume in terms of ad spend and SQL Pipeline. When trying to run causal models, the more volume we are driving the easier it will be to identify statistically significant changes. Sample size and effect size work hand in hand at allowing us to uncover the impact of our ad spend and make meaningful decisions with our budget allocations.

How much sales (online and offline) did each media channel drive?

What was the ROI of each marketing channel?

How should I allocate budget by channel so as to maximize my KPIs?

Where should my next marketing dollar go?

What is the optimal level of spend for each major marketing channel?

How would sales be impacted if I made X change to my marketing plan?

If I needed to cut my marketing budget by X%, where should the dollars come from?

How is performance of channels such as FB impacted by the way they are executed (e.g., buying objective, frequency, creative quality or targeting strategy used)?

What is the impact of competitor advertising on the performance of our brands?

How much incremental revenue to trade and promotional activities drive?

The question(s) we try to answer will shape the data we need to gather to feed into the model.

Determine the question(s) that the business is trying to answer with MMM.

Decide within the business whether relying on a 3rd party tool or an in-house build would be appropriate for a MMM.

There are a lot of vendors that could help fill this gap, along with self-serve tools like 

Research will be a key factor when determining the right path here.

Based on our answers to both the above, there will be requirements on data and Martech. These include (and are not limited to):

Assessing our data needs and understanding if there are gaps in our data availability for any given channel (more to come here).

Developing out data pipelines to automate the analysis and output from a MMM (whether a 3rd party tool or internal build).

Building the initial infrastructure for MMM: ingestion of data, modeling of data, and sharing results.

Develop frameworks and a clear schedule for experimentation to fully utilize MMM as a decision maker.

Iterate and work in a collaborative way across Data, Martech, and Growth to make decisions based on the output of this tool.

This is a less complex, but important and powerful tool in marketing attribution.

At its core it’s just a simple survey that allows respondents to tell us how they heard about us.

The complex part is that how they heard about us may not always match the technical, analytical attribution.

Additional complexity arises with the nature of a SaaS business model. As the number of folks required to make a final decision on adopting a new SaaS product increases, does the answer to a single survey provide us the insight we need to drive decisions for Growth.

Ex: Person comes from a site with a utm_channel=email, but reports that they head about us from TV. It leads to interesting questions:

Were these touches reported and seen in our attribution model?

Did the HDYHAU channel create a halo effect that drove the actual attribution.

Notes From [redacted]: He’d like for us to investigate the following …

This can help us start to better understand the 'halo' effect that paid ads provides to the rest of the company.

"X% of leads say they heard about us from Facebook/Twitter/etc.".

Are we logging both client side and server side events?

Are we stamping the HDYHAU fields to the profile for conveyance to the rest of the stack - particularly SFDC ala other incrementality attributes.

Data team can evaluate current output from logged events / answers to the survey.

High level analysis of how often the survey is answered?

Overlap with the current set of goals / KPI?

Data cleaning, if necessary, to align with the current channel buckets

How does this align with attribution (FTA, Linear, etc.)?

From this initial analysis, we can choose to go several routes:

Revamp / optimize our survey. Including discussions on what point in the user journey do we ask this question, what questions should we ask, etc.

Understand how to apply these results to other attribution outputs? This is key as analysis from HDYHAU is usually combined with another measurement framework (or many).

Utilize survey results to generate multipliers on top of our attribution data (MTA).

Request for Comment on Mixed Media Modeling and How Did You Hear About Us by Austin Hay

RFC: MMM & HDYHAU at Financial Automation Platform

What are we trying to accomplish with this doc?

Prefix

The Attribution Stack

Sign up to see more