These conversations typically lasted about an hour, but some went as long as 90 minutes. When a user was getting really into the exercise and pulling in different design elements, or illustrating what they liked about other tools, we didn't want to cut them off.


The unique thing about working on ads at Facebook at the time was how much visibility we had into how people used the product. I could tell exactly how much money a customer was spending and what their ads looked like, and get familiar with all of their campaigns before interviewing them. It’s one of the only products in the world that I know of where you have that visibility into your customer behaviors, which changes how you do research.

A trap a lot of researchers fall into is to skip the understanding of how users do the work. We spent a lot of time unpacking their process because it was going to be important for us to understand why the thing they drew for us later made sense for their work.

One of the failure modes I see folks run into when they first start writing discussion guides or planning for interviews is they treat it like a survey that they’re reading out loud. Very early in my career, I started writing extra niceties and acknowledgments of gratitude very explicitly into my scripts so that I got in the habit of adding them in. Those things are almost like punctuation in an interview, but it's important to do them in a way that's respectful or acknowledging of the time and information they’re giving. I think it's an important thing to do to acknowledge the people you're talking to.

When you're doing this kind of work, it’s important to set expectations around how we're going to spend the time. In all forms of research, we explain that “There's no right answer, we're here to hear your feedback, etc.” But when you're doing a design activity, that becomes even more critical because you really want them to leave the existing product out of it. We spend more time than average on this upfront context-setting part to really give them permission and empower them to get creative.

You can imagine two separate curves over the course of an interview. The first is trust, which goes up over time as you build rapport. The other curve is anxiety, which spikes at the beginning and then goes down over time. The peak of the conversation happens at that sweet spot when anxiety is low and trust is high.

One of the challenges of doing this kind of work is you have no idea what you'll get. If the output was pretty vanilla, we had a set of backup questions to probe further. But in general, these users had really strong opinions about what was important vs. not important.

The tougher conversations were when users said they just wanted to export the data to their own tool. The joke was that we were actually competing with Excel. But we remained agile with our questions to get a better sense of, given that they think Excel is their solution, how can we learn why so that we can go and make good decisions about our product?

Behzod Sirjani

Leading participatory interviews requires flexibility, preparation, and empathy. I designed this script to enable us to capture the right insights from users about how they worked.

Facebook Audience Insights Team - Script for 

Thank you for joining us today! We're excited to dig into your planning process and have you share some ideas for how Facebook could help you do that better.

In addition to asking you some questions, we're going to invite you to doodle or loosely sketch some ideas out for us. The purpose of this is helping us better understand what you'd like to see or use in a product - it's not a design critique at all.

Before we dive in, we would love to record this conversation and take some photos of the things that you draw. I'll ask you again at the end of the session if you're okay with us keeping these things, but for now, are you okay with us recording as we go?

To start things off, I'd love to learn more about you in your role as a [Media Planner]. What are some of your primary responsibilities at [COMPANY]?

Note to self: For each responsibility, dig into who they work with to accomplish this, how often these things happen, and the primary tools and services involved.

Note to self: Try to get a sense of their planning cadence and tooling.

JANE: Media planner at Acme Bike Co, they sell "college commuter" type bikes (sub $1k price, either single speed or 9 gear). If she does her job well, more people know about (and buy) Acme Bikes. She owns audience definition for each seasonal campaign as well as channel selection and KPIs. Primarily focuses on Brand measurement because they sell through retailers.

JANE: Has a team of 4 people, one who oversees creative, one for copy writing, two who help with trafficking and buying - team works as a unit, traffickers split work by platform (one for FB/IG, the other for YouTube + other platforms)

What are some of the most important things that you do each week?

Note to self: If they are struggling to answer this question, ask "If you didn't show up to the office for a week, what would go wrong or fall through the cracks?"

Pick one of these depending on the frequency of their campaigns.)

JANE: She monitors campaigns weekly, adjusts spend at the same cadence. Typically runs campaigns for 4-6 weeks, refreshing creative between campaigns.

JANE: On a longer timeline, campaign development is the bigger priority. She works with analytics and retail teams to understand what's selling well, develops new positioning, and helps build collateral for retail partners.

Observation: This is a unique situation because they don't sell the bikes themselves, they drive traffic to retailers. It means she has to balance the brand audience with the retailer audience and probably relies on other data. We have to figure out how they do that.

Now I'd love to dig into your planning process. To do that, can you walk me through a recent campaign you worked on and talk about how you planned for it from start to finish? 

How did they determine these audiences? What tools were used? What past data?

What channels did they plan to advertise on? How did this affect planning and audience definition?

How were they going to be evaluating success?

JANE: Fall 2020 they ran a "no peloton no problem" campaign to encourage people to enjoy the outdoors, creative was focused on young adults who probably couldn't afford a peloton but enjoyed being outdoors.

Primary objective was raise awareness and drive traffic to retail partners

Primary audience was young adults with sub $100K income who live alone or with roommates

This was based on retailer purchase data and what they know about the peloton customer

Secondary audience was adults who live close to work and also make sub $100k - they know that people use their bikes for short distance commuting, which isn't common in pandemic, but could be additional purchases

Most of this was based on behaviors they saw from retail partners and new newsletter sign ups, plus industry research on consumer spending in 2020

Most of advertising was FB and IG. They don't have a big video budget, typically one per year, more of an evergreen thing

They spend ~ $10k per month across all channels

Primary metrics were engagement (likes, follows)

. My last question before I shift us over to the activity is about the FB Audience Insights tool. How familiar are you with the Audience Insights tool?

Where in your planning process has it been most helpful?

Where have you tried to use the Audience Insights tool, but found it not as helpful?

How does it compare to other tools you use for planning?

JANE: AI is great because their primary platform is FB - helps to do the planning in the tool you buy in.

JANE: Typically uses AI a few times a year, but will often try to make budgets maximize reach, so her traffickers have ability to be loose with audience criteria to maximize reach

JANE: No real other tools for planning, aside from internal reports from past campaigns 

Observation: This actually splits audience definition into two phases - one is when she does it and another is when her team creates the ads. Seems like a data disconnect for them.

Great - now I want to shift gears and move into an activity for us.

What we're going to do is have you draw out what some of your ideal planning tools look like. The goal is not to design a perfect UI, but for you to call out what metrics, charts, or other information can be helpful for you.

There is absolutely no right or wrong answer here, we're doing this to better understand how you think about planning and the way we can support that process.

As you're going along, if you have any questions or are confused, please let me know.

Here's a piece of paper and a marker. What I would love for you to do first is imagine that you're starting to plan the campaign we talked about earlier (

be specific about the one that was discussed).

 In your ideal world, when you open up your planning tool, think about what you would see. What data do you have access to? What format is that data? Now, I want you to take 5 minutes or so and sketch it out - in enough detail that we can talk about it. I don't have to know what it is, but I want you to.

Note: Let them know 1 mins before wrap-up, and give 1-2 extra minutes if they need it

 I'd love to start us off with you walking me through what you've drawn. 

Note: Give them time to walk through everything at a high level. Listen to where they focus and what they are calling out as important.

JANE: Drawing is pretty basic because most of what she wants is basic demographics but the ability to cut by brand engagement/saved audiences.

JANE: Primary metrics are demographics with all data split by everyone, primary audience, secondary audience. For most of this, she wants to see charts because she wants to make comparisons rather than know a total.

JANE: After primary metrics, basic engagement info is helpful (what % of people who followed are age X-Y, who has clicked or commented, etc)

UP: This is fantastic. I'd love to return to [first thing they mentioned.] Can you tell me more about why this [metric/graph] is the thing you want at the top of the of the interface?

What kind of decisions does this help you make?

JANE: Knowing how on target they are is the main thing. They're trying to drive awareness, so they want to make sure we're spending money on the right people.

How do you think about this [metric/graph] in relation to the ones around it? 

Note: You want to understand if this is valuable on its own or a piece of a broader story.

JANE: Seeing the amount of people in their target demographic who have engaged versus not is key because it helps them understand where they are hitting saturation versus could spend more money. Because they have pre-established audiences, seeing engaged/non-engaged at a demo level helps them think about how to better target next time.

Why did you choose for this to be a [metric/graph] rather than a [graph/metric]? 

Note: they may want to be able to toggle between them. Ask!

JANE: The metrics on their own are a lot less helpful - we want to see them as a comparison. A table would be okay, but chart is easiest to read, ideally bar chart so you can see audiences split by color

For graphs, dig into how they want the information visualized and what that visual allows them to do.

UC: If this [metric/graph] wasn't here right now, what other product would you use to find it?

They would download to excel and make the graph themselves.

UP: Let's continue. Tell me more about this [metric/graph] that you have below it... (

repeat for most of the elements on the page).

UC: Now I want to zoom out a little bit. There's definitely some information I thought you would include here that you didn't. Can you tell me about some of the metrics you might get from other tools or products that you don't need to see included here?

JANE: Seeing retailer engagement or even location data for people who are near a retail partner would be great, but that seems really difficult at scale, and we don't want to worry about the complexity.

UC: Similarly, what are some metrics/graphs that other products include that you feel like are wasting your screen real estate?

Thank you so much for all of the feedback today and sharing your ideas with us. I want to wrap up by returning to you conversation about planning generally. When you talked about Fall 2020 campaign, did you do the audience definition all at once, or was that more of an iterative process as you looked at the data?

JANE: Planning tends to be kind of one-off since she lets her traffickers have some influence over audience.

To what extent is that consistent across campaigns?

JANE: Occasionally they will experiment with new categories/targeting criteria in Facebook, but they tend to have a good idea of what matters on a consistent basis.

Finally, now that you've shared all of this with us, I wanted to ask if we could keep what we've gathered - both the recordings and the images - to share within our organization?

Audience Insights Script at Facebook

Facebook Audience Insights Team - Script for Design Method Discussion

Notes - Participant 1 (Jane)

Rapport

Design Method Build

Getting Context

Sign up to see more