This is phase 2 of an end-to-end product development process at Nuvocargo. 

Check out the blank template for this phase here.

The Kryptonite of product reviews, especially when you get to P2 and people feel really passionately and are really engaging with the solution, is that it's very easy for them to go off the rails, and all of a sudden, 10 different conversations are happening and you're rehashing what happened two weeks ago. The role of the PM is to basically be able to interfere and say, hey, I don't think it's the best use of our time to discuss this right now. I'll commit to a follow up item to address this feedback and make sure to create the right forum to discuss it. 

That can be scary, especially when the approvers are the heads of the company, but ultimately, your job as the PM is to move the conversation towards where you need the input and the feedback, so sometimes you need to do the hard stop. And if it becomes apparent that we're running out of time, that means that we probably need to do another turn on this and I need more input from the heads before we actually propose the P2 solution.

I think of Phase 2 as the nitty gritty. This is what I'm going to be reviewing with the dev team once we actually kick off executing the project. This is the foundation and the roadmap of how we're going to build the solution. 

This project is different from the projects I'm working on now because the P2 is much shorter than the P1, which actually had to go super deep. But because this project was really about building a foundational measurement of one side of our marketplace, most of the work went into aligning on how we wanted to approach building that. And then the P2 was just about providing folks the detail necessary to understand what's included in every release? What do we need to instrument in each release? And how will we know we can successfully move forward to the next release?

I ended up changing the score weights because of the conversation we had in the P2 review because when everyone saw it all on paper, we realized we should be giving more weight to reliability once we were able to start measuring it. And I hadn't considered that fully before. It was a compelling conversation to have in the moment. 

That's the kind of feedback that is helpful in P2 reviews: you're not debating, you're not rehashing whether we should be doing it this way, but instead you're using these minds to refine how you're actually going to tackle the solution. Getting that feedback now versus once I've built the scorecard was a good outcome.

Whenever I kick off any review for whatever phase, I start by setting the context and saying "Hey, we're doing this review today. To remind you, the goal of the P2 review is to align on how we're actually implementing this solution. What success looks like for today is XYZ." For this project, success was making sure we were aligned on what metrics were going to be included in each release of the carrier scorecard and addressing any questions or concerns around how we were getting those metrics and how reliable they were. 

In this case, the two approvers were very close to the operation and had a strong point of view, so we were able to align in the meeting. That's not to say that it's always perfect and everybody always stays on task! The way I manage it when disagreements inevitably happen is by interrupting gracefully and saying, "I hear what you're both saying, and what's important is that we agree in general that this metric should have more weight than this metric. If we agree that metric A carries more weight than metric B, I'll make sure to follow up with you all with, examples of what it looks like if we do it at 30% versus at 25%.”

Anaid Chacon

Isa Oliveres

After aligning all my stakeholders, I dove into the nitty gritty: defining the foundation and the roadmap for how the team would build the Carrier Scorecard.

 Solution Requirements - Carrier Scorecard

Approver: Head of Carrier Team, Head of Marketplace 

Collaborators: Marketplace Analyst, Carrier Team Analyst, Lead Product Manager, Head of Finance, Data Team

The doc should address each of these questions:

What is the complete set of requirements? What cases are we focusing on?

For each must-have requirement, what exactly does this requirement entail? For projects that have user-facing UI, this means providing detailed mocks or prototypes driven by the Product designer assigned to the project.

What's the priority for different requirements based on the goals of the project?

Scorecard Qualified Carrier: a carrier who meets the requirements to have a scorecard (having moved 5 number of loads with Nuvo)

of the Carrier Scorecard Metrics that provides an overall quality score for that carrier

The Carrier Scorecard will be implemented in a Google Sheet

Internal users can view Carrier Scorecard Metrics

Internal users can view computed Carrier Score

Internal users can filter by Carrier to see that Carrier’s scorecard

The Scorecard Qualified Carrier will receive an email with their Scorecard Metrics at a recurring cadence

The Scorecard’s data robustness will be incrementally enhanced with every release

In order for a Carrier to have a scorecard, they must have moved 5 loads with Nuvo

Metric must have at least 5 entries to be included in Carrier’s scorecard (i.e. once we record hour-level pickup data, there must be at least 5 entries with hour-level data to include hour-level pickup rate)

The scorecard is calculated on a rolling basis:

for whichever volume is greater between the last 30 days or last 25 loads (release 2+)

For each requirement that could fundamentally affect the engineering implementation, what exactly does this requirement entail?

For projects that have user-facing UI, this means providing detailed mocks.

Prototypes can be refined as requirements get.

Mock scorecard with mock data: [redacted link]

At a high level, how will this work technically? What is the confidence level in the engineering feasibility of this project? 

This portion should be worked on collaboratively by the PM and TL/key engineer.

What, if any, prototypes have been undertaken to increase confidence in the engineering feasibility?

This doesn't need a full eng implementation plan.

TL suggestion: Skip a level and use metabase to display results. It allows power users to create their own queries and allows simple filters to be added on top of the results. It also eliminates the input delay caused by syncing PG to Google Sheets and the need to create a script for said sync.

, it includes a carrier ID, a metric name, a metric value (numeric integer), a metric sample date, and probably a leg_id. For 

, we need to create some sort of status column to account for bounced legs, a timestamp for when that occurs and stop deleting legs when the bounce happens, they should be readonly from now on.

Every time a carrier performs one of the actions that is relevant for a metric we calculate the new metric and create an entry into the 

Query and group by carrier and metric type. (Advantage: we can ask “which carrier is the most punctual?”, “what is the average time carriers take to upload their PODs?”).

A single table for SQL-versed people to query when they need answers.

Query efficiency and consolidation, no need to know about the complete data structure to get answers.

This plan is feasible to be carried out by a single back-end leaning engineer (or a group of them).

No prototypes have been created to make this decision.

Data is queried via SQL and surfaced via Google Sheet

Carrier Scorecard Google Sheet data is refreshed on a daily basis at X time

TL to fill in bounce rate instrumentation details

TL to fill in scorecard table creation plan

If everything goes according to plan no further DB changes should be required.

What are all the edge cases that must be addressed?

Are there any notable edge cases that do not have to be addressed? Why?

What are the known limitations of this approach and why are we comfortable with them? Are internal customer teams comfortable with them?

Opting for a lean scorecard with just three metrics requires slightly more complexity in the calculation of the metrics (i.e. having multiple inputs) - internal teams are ok with this as long as we clearly communicate what those inputs are and how the metrics are calculated

Enhancing the robustness of scorecard metrics with each release will require additional training and communication. Internal teams are comfortable with this since there’s urgency to begin getting visibility and agreement that iteration is an important part of the process.

Solution Requirements: Carrier Scorecard at Nuvocargo

Phase 2: Solution Requirements - Carrier Scorecard

Requirements and Use Cases

Sign up to see more