This is phase 1 of an end-to-end product development process at Nuvocargo. 

Check out the blank template for this phase here.

Approvers depend on the project, but it will generally be the head of product and the operational heads responsible for this area. So, in this case, it was the head of the Carrier team since we would be measuring the performance of carriers and the head of the Marketplace team. I think that's a little bit unique to a company like ours, and also what makes the product development process extra helpful here. When you're dealing with many different approvers, it forces you to think about every vector and integrate all of it into the solution you're proposing.

Phase 1 tends to be the hardest because it forces you to push yourself to be principle-driven and understand, what are the right axes upon which to compare solutions? And that is the brain-crunching work that can be really hard to get right. But it's important because what you're really trying to do in Phase 1 is outline the tradeoffs of the different ways you could solve this problem. Especially when you're dealing with an audience that's not exclusively engineering product and design, what makes a Phase 1 review successful is if you've nailed some clear and differentiated axes upon which you're comparing the solution and you're able to explain the tradeoffs. It should be a tradeoff-driven conversation, so instead of debating, "Should we be implementing this in a Google sheet versus a looker?" you're actually debating, "What's the tradeoff between those two things?"

I think it goes wrong when everybody identifies what tradeoff is most relevant to their day-to-day work and just rabbit-holes down it. So if the head of sales sees that one of the tradeoffs is that it'll be harder to get sales tax information, for example, they only want to have that conversation. And then meanwhile, the head of engineering is talking about how there are a lot of dependencies and rework on trying to backfill this data, and everybody is speaking a different language. Whereas anchoring in tradeoffs means everybody is speaking the same language, and that makes for a productive discussion.

I reached out to this guy on LinkedIn (love it when people respond!) who had built the carrier scorecard at Uber Freight, which is a much more mature company than we are. And in this conversation, he shared, "Keep it simple. Don't overcomplicate things. Everybody's always going to tell you there's not enough data. And you really need to start with the basics, move forward, and make it better each time." 

That struck a chord with me because even in the initial conversations I had internally, there was so much noise about "We can't trust our data, this is not reliable, and we need to be measuring all these minute details." And this external validation gave me the conviction that starting with the basics would be key to making this project successful and actually map back to the narrow scope from the P0.

The dynamic inputs piece came about in the conversations I was having with the head of the carrier team, who was obviously concerned that if we were scoring carriers all of a sudden, it might turn out everybody had a bad grade. So if we truly believe that the purpose of measuring carrier performance is that we're able to give them feedback and incentivize them to be better, then we need to only measure the things that they can control. That felt like the fairest way to approach this. 

None of the proposed solutions are measuring 87,000 things and building out a robust product experience. But if we didn't have those initial principles, the P1 conversation could have been like, "Why haven't you considered all these other things?" Establishing this upfront helps people understand why this is the option set.

In phase one, I'm not thinking about the exact metrics in the scorecard and where that data is coming from. I'm just thinking, what are all the ways we could build a scorecard? And how are they different? So the goal of P1 is that I have been expansive enough in exploring different solutions based on my guiding principles that we feel confident we're moving forward with a good decision. I don't like to say the best decision because a lot of things could be the best. But it's a good decision, and we're aligned on moving forward with it.

I did get some pushback, specifically around the sequencing of the releases. It went back to the feeling of "We don't have the right data, we don't have enough data, I don't trust the data, and therefore I don't think it's fair to measure and potentially give somebody a bad grade on insufficient data." And on the other side of the equation, folks were saying, "We need something now, even if it's just an on-time score for carriers." Anchoring on the speed component allowed me to get folks to align on this.

By opting for a low-fi and easy-to-implement solution, we could invest engineering resources in improving the fidelity of the data versus building a UI. And the other helpful piece was framing this iterative release as, "Hey, this isn't just to move fast. It's so that product and operations can jointly use the feedback." At a company like ours, where we're still figuring out the best way for product and operations to work together, viewing this solution as a way to build a tighter feedback loop was compelling.

What you wouldn't know about Phase 1 was that I had originally assumed that the project's scope was purely internal. But when I shared the P1 for an early review with the head of product, Anaid's first comment was, "What do you mean we're not sharing this externally, and there's no plan to do so? How will carriers know they need to perform better if they don't know what is expected?" And I remember reading that and thinking, "Oh no, this is crazy. This expands the scope so much, I'm gonna have to realign everybody." 

But Anaid reframed it by saying, "My feedback isn't that we need to do a big public launch. But you won't be able to move the metrics you want if there's no path to this being externally facing." She challenged me to create a readiness plan for a very lightweight rollout, like the rep that manages the carrier sending them an email. Once we'd had that conversation, I was able to integrate that user interface external piece that wasn't there before. That ultimately became essential to how we thought through the rest of the project.

Anaid Chacon

Isa Oliveres

By starting with the basics and anchoring on tradeoffs, I was able to align a highly cross-functional set of approvers and collaborators on my proposed solution.

Head of Carrier Team, Head of Marketplace Team

Collaborators: Carrier Team Analyst, Market Place Analyst, Lead Product Manager, Head of Finance, BizOps Manager, Data Team 

Informed: Head of Operations, EPDM Meeting Notes, Leadership

Problem Definition and Key Characteristics of the Solution

, we currently don’t have a way of programmatically assessing the quality of our carriers, and we will address this by creating a Carrier Scorecard. This Carrier Scorecard will allow us to:

Measure carrier performance across a consistent set of attributes

Calculate a “Carrier Score” to assess carriers’ overall performance and compare among them

Give carriers feedback to improve their service and reliability

Use scorecard metrics as inputs to other initiatives, such as improving shipper<>carrier matching or providing operational performance data to shippers

Increase in weekly revenue retention for shippers (by helping us improve the service quality they receive)

Improvement in scorecard metrics (e.g. on-time pickup rates and POD upload speed)

Product adoption and engagement (carriers will find a benefit to taking actions on our platform, which will in turn reduce our cost to serve)

All-in net revenue (through improved matching of carriers and reduced cost to serve)

Dimensions of the Solution and Principles

What are the key axes along which potential options differ? 

Are there unavoidable tradeoffs between these axes? (ex. fast, good, cheap - pick two)

For each option, common axes to consider are:

Estimate of engineering (and other) resources needed

Monetary cost if substantial (ex. additional servers needed, software licenses)

Additionally, there will frequently be at least one axis that is unique to the project

The guiding principles for the Carrier scorecard are:

 - the scorecard should contain the minimum number of attributes necessary to asses carrier quality

“You have to keep it simple - don’t overcomplicate it with too many metrics” - Head of Carrier Strategy at Uber Freight who led their scorecard development and roll-out

- we will score the carrier on the behaviors they can control. Static inputs (i.e. fleet size, preferred routes) are captured through carrier profiling and can be used in conjunction with carrier scorecard metrics separately.

Based on these principles, the first version of the Carrier Scorecard should measure service quality and product engagement. 

The scorecard consists of three top-level metrics - On-time, Reliability, and Engagement - some of which have multiple inputs:

The way in which metric inputs are used to calculate top-level metrics and how the scorecard metrics will be used to compute a “Carrier Score” are outlined in the P2.

The key differences between the proposed solutions are the completeness/granularity of data required for roll-out and implementation complexity. The main trade-offs across these axes are speed and resources: The greater data completeness required for roll-out and the more complex the implementation, the more resources will be required, and the longer it will likely take to get the Carrier Scorecard out.

What are the strengths and weaknesses of this option? 

If this project has substantial technical complexity, include technical strengths and weaknesses.

Make sure to touch on whether there will be any deployment complexity (glance through phase 3 questions)

What are the security and privacy implications of this option, if any?

What are some of the other options, what are their strengths and weakness, and why are they not the best option?

If there are internal customers of this project, do they also prefer this option?

If not, why is this still the best option? (ex. future, currently non-existent customers, will benefit substantially)

The potential solutions encompass any combination of the three attributes (Data completeness, Data accessibility, and User Interface) across the implementation complexity axis. The attributes are independent from one another: For example:

We can opt for full data completeness implemented in the least complex manner (a unidirectional Google Sheet that is shared through screenshots in a manual email).

We can opt for full data completeness implemented in the least complex manner (a unidirectional Google Sheet that is shared through screenshots in a manual email)

Each combination of attributes presents the trade-offs outlined above: low implementation complexity translates to greater speed but less robust data inputs at the outset, and less scalability for other teams to take advantage of scorecard data.

What is the best option? Why will it best achieve the success criteria from Phase 0?

The proposed approach is to sequence several releases

 that build towards full data completeness and multidirectional accessibility while maintaining a simple user interface in Google Sheets. This solution best allows us to meet the success criteria of Phase 0 by:

Prioritizing speed to roll-out so the Carrier Team can get initial visibility and begin to address pressing issues

Allowing other teams to use scorecard metrics as inputs to their own initiatives from the get-go by having scorecard data be accessible through Google Sheets, where the majority of scoped initiatives will be implemented at first.

Providing the opportunity for internal users to give feedback early and often as we invest in improvements with every release

Internal teams prefer this option - the Carrier and Customer Success teams are eager to have an initial scorecard in place to start addressing key service issues. Parallelizing the dev work required to make the scorecard more robust will ensure that work is properly resourced, while still allowing other stakeholders (like Marketplace) to be able to leverage scorecard data in their own initiatives starting with Release 1. There is limited value in increasing the user interface complexity of the Carrier Scorecard at this stage - it would require training internal team members on a new tool and would increase the cost of rolling it out.

 will encompass partial data completeness with unidirectional accessibility, while the dev team will work in parallel to begin instrumenting additional/updated data. At this stage, we will begin training internal users on what the scorecard is and how it can help them, but will not yet share it externally:

 will integrate more enhanced data into the existing implementation and the scorecard will begin to be 

 will incorporate all v1 metrics and extend data accessibility to be multidirectional. The multidirectional data accessibility will allow us to then automate the process of sharing the scorecard with Carriers via email:

Solution Proposal: Carrier Scorecard at Nuvocargo

Phase 1: Solution Proposal - Carrier Scorecard

Problem Definition and Key Characteristics of the Solution

Sign up to see more