Marketplace Experimentation Can’t Be One-Sided

Experimentation is a core competency for product managers across any type of product or company. But for product managers in marketplaces, experimenting with new ideas is especially challenging. There are two different customers that need to come together to make a marketplace work — the demand-side and the supply-side — and one side always impacts the other.

Further complicating things, marketplace product teams are often organized according to which side they serve. For example, at a ride-share marketplace, there will be separate teams focused on drivers, the supply-side, and riders, the demand-side. At a home services marketplace, there will be teams focused on service providers and teams focused on the homeowners who need things fixed.

In these situations, it’s common to experiment with new ideas in silos, and usually the demand side of a marketplace experiments more than the supply side due to a larger volume of traffic or users. The problem is that features on the demand side can have significant repercussions on suppliers if fully launched—and too often, those repercussions get overlooked before launch.

Think about a figure skating pair. Each skater can experiment with style, facial expressions, and maybe minor footwork without impacting the other — but if the skater in the air decides to change their foot position mid-flight, that could mean a skate to the face for their partner waiting to catch them.

Phil Farhi, former Chief Product Officer at Thumbtack, and Adam Grenier, VP of Growth at Eventbrite, have both experienced “skate to the face” moments with demand-side product managers and have had to react quickly to stop a feature from being detrimental to suppliers. They’ve come to realize that experimentation in a marketplace is unique, and requires partnership between demand-side and supply-side product managers.

In this article, they’ll discuss:

• How to improve your pre-experiment approach to experimentation as a demand-side PM
• How to evaluate the potential supply-side impact of a demand-side feature
• What to do with each type of potential supply-side impact

Meet The Contributors

Demand-Side Experiments Require Supply-Side Involvement

Before any experiment on the demand side of a marketplace begins, it is important to anticipate the potential impact of a demand feature on the supply side of the marketplace.

“At the experimentation phase, you don’t need to have solutions for all the effects a demand-side feature might have on supply. However, you do need to have an idea of what those effects are, so you can plan your experimentation approach and analysis effectively.”



— Phil Farhi, former CPO at Thumbtack

So demand-side product managers need to ask, “What will happen to suppliers if this feature is fully rolled out? And how will that impact evolve over time?”

Often when the demand-side attempts to do this exercise on their own, they underestimate the problem or jump to an uninformed solution. This exercise is critical for both supply-side and demand-side product managers to do in partnership.

This collaboration may feel clunky at first, but it is worth the investment. With each experiment over time, both sides will develop a shared understanding and deeper empathy for each other’s needs.

The Risk-to-Supply Spectrum

To facilitate this process, Adam and Phil developed a framework called the Risk-to-Supply Spectrum.

The Risk-to-Supply Spectrum helps demand-side and supply-side product managers to work together to determine the potential supply-side impact of a demand-side feature and what actions both sides need to take in order to test the feature effectively.

For supply-side product managers, this provides them with the ability to prepare for potential demand-side features and to help their demand-side counterparts think more about impacts on suppliers. For demand-side product managers, this enables them to choose the right experiment that provides the entire product team with the data needed to make a decision on how to move forward.

The Risk-to-Supply Spectrum is a four-part spectrum that is based on the potential impact a demand-side feature is expected to have on suppliers. After product managers assess the potential supply-side impact, the color coding provides guidance on what type of experiment to run and how much resourcing the supply-side should prepare for a potential feature roll-out.

Assessing the Risk of a Demand-Side Experiment

The first step to use the Risk-to-Supply Spectrum is to determine the potential supply-side impact of the demand-side feature. This should be a collaborative exercise with product managers from both sides of the marketplace; the demand-side product manager brings details on the feature, and the supply-side product manager brings expertise and empathy around suppliers.

There are two questions that inform where on the spectrum an impact falls:

• Who do you think will be impacted? The more important this group is to the business, the more you move toward the red side of the spectrum.
• How severe do you expect the impact to be? There are two factors that would cause you to move from red to green: impacting a part of the supplier experience that is critical and having a high frequency of impact.

Question #1: Who is impacted?

The most common mistake Phil and Adam see product managers make is assuming an average impact across a marketplace.

While some demand-side features do impact the entire supplier base, the most important consequences to identify are when the potential impact is on a critical sub-group of your suppliers. To avoid this, product managers should identify what characteristics of suppliers might make them more impacted by a specific feature.

For example, at Thumbtack, it is important to understand if the demand-side feature will impact the group of professionals who use Thumbtack as supplementary income versus others that seem to use it for their core business. On a hotel booking platform, the team might want to consider if there will be an impact on chains or independent hotels, or on a particular hotel or hotel group that provides a high daily average of rooms.

The key to this step is to then identify which sub-groups of suppliers require the most risk mitigation. The more important that group of suppliers is to your business, the more towards the red side of the spectrum you move.

It’s possible that a feature will impact an objectively large number of suppliers, but not the suppliers that are most critical to the evolution of your strategy. In that case, a marketplace might care much less about who is impacted.

On the other hand, there might be a very small group of suppliers impacted, but those suppliers are absolutely critical. Often, those are suppliers that provide a large portion of the supply units. In our figure skating example, you can think of the ankle of the partner who catches the flying skater. The ankle is a very small part of the body, but the catch will be a disaster if the ankle is unstable.

Question #2: How severe is the impact?

The severity of the impact will be a function of what part of the supplier experience is impacted and how frequently it is impacted. The more you are impacting a critical part of the supplier experience and the more frequently suppliers feel an impact, the more towards the red side you move.

For a professional at Thumbtack that gets 1-2 leads from the marketplace each month, losing one of those leads is very severe. But for professionals who get 15 or so leads a month, losing one of those leads might not be severe.

For a small hotel, uploading rooms to a travel marketplace manually, so increasing the time it takes to upload a room might be detrimental. For hotels that can do batch updates, adding an extra field might not be a big deal. For a driver in a rideshare app, anything that impacts the most stressful part of the ride, pick up or drop off, will happen frequently and targets an already stressful moment.

With these two questions, supply-side product managers can then assign a potential impact to a color: green, yellow, orange, or red. These questions and this process work across all marketplaces, but the nature of how expensive it is to acquire suppliers might accelerate how quickly an impact moves towards the red side of the spectrum.

“If your supplier costs you $2,000 to acquire, you might have a different perspective on what severe means than if your supplier costs you $500,000. The more expensive the supplier is, the more risk averse you want to be.”



– Adam Grenier, VP Growth at Eventbrite

Determining the Right Experiment and Resourcing Plan

The type of experiment that demand should run and the right resourcing plan for supply will change based on the spectrum risk level.

As a general principle, as the potential impact of the demand feature on the supply-side moves from green to red,

• The experiment should become more controlled and have clearer supply-side signals.
• The more capacity and resourcing the supply-side should prepare in order to support the demand-side feature should it have positive results in experimentation.

When the Risk to Supply is Green

Green means one of two things:

• The suppliers impacted are not critical to current and future strategy
• The impact of the demand-side feature on the suppliers is, at worst, very minor

The most common examples of “green” demand-side features are ones that open up new growth channels. While new demand might not always be equally distributed, there’s rarely a first-order impact that is major enough to reprioritize supply-side resourcing.

These features are easiest in terms of finding a mutually beneficial path forward. The demand-side should have the freedom to run any experiment type to prove the value for the demand-side. Unless the results are unexpected, the supply-side team should not plan any specific resourcing before the experiment begins.

When Risk to Supply is Yellow

Yellow means the marketplace expects suppliers that are critical to the current or future of the business will be impacted. However, the severity of the impact is unclear and needs to be measured.

For example, at Thumbtack, the demand-side wanted to test a feature that encouraged homeowners to reach out to additional professionals after reaching out to the first one that they chose.

On average, they expected the feature to grow the number of jobs that were ultimately completed, a net positive to the marketplace. However, from the perspective of the professionals, any given lead now might become more competitive.

The supply-side team thought the potential impact on the professionals who won 1-2 leads per month might feel pretty big. If enough professionals lost one of those leads, they would feel like they are losing a majority of their chances to earn thanks to Thumbtack, monthly. This population of suppliers was significant for Thumbtack, but they did not yet know if the feature was going to impact a small or large part of that subpopulation.

This scenario is where there is the most flexibility in the type of experiment. Because there are many possible types of experiments, the supply-side and demand-side need to work very closely to make sure that the experiment that is run provides results in a way that lets them understand the severity of impact. Depending on the severity of the impact, the supply-side product manager will have to determine if they need to build a supply side solution.

At Thumbtack, when the experiment results came back, the team saw a high portion of those professionals experience a decreased value per lead, and so the team knew they had to have a solution to help mitigate the impact before the demand-side fully rolled out the feature.

The supply-side of the product team needs to ensure they have the resourcing available to work with the demand-side product manager to set up the experiment in a way that provides them with useful results. They also need to keep a bit of flexibility in their roadmap to accommodate building a supply-side solution should the results demonstrate high severity.

When Risk to Supply is Orange

Orange is when the supply-side knows that the suppliers impacted are critical to the business and that the impact will be severe.

For example, at Uber, when the demand-side team wanted to roll out a change that would help drivers get more rides per hour, the supply-side team realized something like this would reduce the ability to have bathroom breaks without pausing the application.

While that likely wouldn’t impact drivers who only complete a few drives a week, this impacts drivers who drive 40+ hours a week with Uber. And those power drivers are critical to the business; they are responsible for most of the completed trips. The supply team knew the flexibility to take breaks was a core part of their Uber experience and they would feel this impact very often, likely multiple times per day.

As a result, this category of impact should have experiments run where the teams can gather strong signals on how suppliers will act. The supply-side team will use this information to start generating ideas on what the best supply-side solution can be. A common test here is a market-based test. This is when you find two separate markets that are comparable, such as two cities, and you launch the feature fully in one to compare results to the one that has no feature.

“These market-based tests might be harder and more expensive to run, but if you think the demand feature is going to have a really big impact, this test is going to give you a strong signal of what is really going on.”



— Phil Farhi

For orange features, the supply-side team needs to be ready to allocate resources and shift the roadmap in order to build a supply-side feature. The results of the experiment should help provide ideas on what types of solutions might be successful because they should have more information on supplier behavior.

When Risk to Supply is Red

Red is when the impact is so severe that it will violate the sense of trust suppliers have in the marketplace, or compromise the differentiating factors that make suppliers choose this marketplace.

If a supply-side product manager is anticipating this impact, both the supply and demand-side product managers need to raise this to the product leader to determine if the experiment should even move forward.

“If the answer to the question ‘Does this violate trust?’ is yes, then don’t do it.”



— Phil Farhi

Another way to think about if the impact to the suppliers is red is to think about if the reaction could put the entire supplier community at risk. For example, at Masterclass, if the demand-side had tested using the talent’s image or information in ways that really impacted the talent’s brand, Masterclass could lose trust in general from the talent that chooses to build classes with them.

“In some situations, you’re not only hurting that specific relationship, but you’re also potentially ruining all the supplier relationships.”



— Adam Grenier

Always Try to Prepare for Supplier Impact

When supply and demand teams inside a marketplace work together, what becomes a solution for one side of the marketplace can help the ideation and progress on the other side of the marketplace. And ultimately, if both sides do not move in lock-step, the concept of a marketplace falls apart.

Keeping the supply-side team informed and involved in demand-side experiments leads to a supply-side team that is able to proactively enable their suppliers to be successful in the market. Evaluating each experiment in a stoplight framework is just the starting point. The supply-side and demand-side product managers can learn from each experiment on what type of features start to fall in each category. This builds overall understanding and empathy for all sides of the marketplace, instead of just the one side represented by the team.

“You take the learnings, you go back to how you’re categorizing features on the Risk-to-Supply spectrum, and see if anything needs to change. This isn’t a static thing you do once and never touch again.”



— Adam Grenier

Reforge has a library of content, including a four-week program about Experimentation & Testing available on-demand. To learn more about this topic and more, apply today!

Apply To Reforge