In Phase 0, I had no idea what the baseline was. Because I didn’t have a good sense of what was achievable, it may have felt like I was sandbagging these metrics. By the time we got to Phase 4, because we’d taken this iterative approach, I took the input of what we were seeing and was able to refine the goals from there. It’s a good learning going forward.

Everybody was really excited about the scorecard, but then people didn't use it immediately. So we actually set up a biweekly call with all of the carrier reps, their manager, and one of our analysts. And the goal of the meeting was both to get feedback from them about the scorecard, and also to encourage them to give their carriers feedback to help them improve. 

With a product like this, the impact is going to be driven by people using the data to inform the decisions they're making. Imposing that structure onto them in a way that's a conversation also allows them to give feedback and helped us improve the accuracy of the scorecard, while also encouraging them to use it in their day-to-day.

This is the kind of thing that illustrates why measuring is important: at least now we have a baseline.

95% came from multiple conversations with the head of the carrier team and the head of operations as we were discussing the path towards releasing this externally. You don't want to launch a product like this externally and then have the experience be bad for the carriers. They wouldn't trust us anymore. So therefore, we needed to really set a high bar for ourselves on this one. Even though we didn't meet this goal after launch, I'm okay with this being red, as long as we have a path towards making it green.

Anaid Chacon

Isa Oliveres

After launching the Carrier Scorecard, my team and I gathered qualitative and quantitative performance data to inform next steps. We deduced that although it was performing well overall, the Scorecard had yet to reach its full potential impact.

Phase 4: Results Readout - Carrier Scorecard

This phase aims to understand how well our solution is performing post-launch and what adjustments/iterations may be required.

☝ This should be shared and reviewed asynchronously and team members should be given at least 24 hours to provide comments / approval.

👉 Following this meeting, please send a list of decisions, next steps, and meeting recording to epdm-meeting-notes@nuvocargo.com. Meeting recording links will be provided by Anaid’s EA.

How has this solution performed (quantitative)?

Driven by data team: Include both high-level success metrics and any other metrics you are actively tracking.

⏱️ We should analyze performance for at least 2 weeks post-launch

Sandbagged the metric because didn't have a baseline

Forcing people to use scorecard through review sessions with managers

At least now we have a baseline

95% completeness of scorecard data

What qualitative feedback have you received?

This could be Zendesk tickets received, team members telling us something is working or not working, general feedback from users etc. Please detail this feedback below.

General excitement around the Carrier Scorecard launch - operating team members quickly started expressing their concern regarding the accuracy of the on-time metrics in the scorecard, since they were not taking into account the appointment type.

We also received feedback asking for additional views/cuts of the data to make it easier to make operational decisions based on the scorecard.

We incorporated this feedback into the v2 of the Carrier Scorecard, which included the following changes:

: the scorecard now takes into account the appointment type (window, exact time, all day) for pickup and drop-off appointments, significantly improving the accuracy of how we’re measuring the carrier’s punctuality. You can find more details on the calculation methodology here [link redacted]. Huge thanks to all the Carrier Reps who filled in the appointment types - we’ll further enhance this once we build in appointment windows and loading procedures into the platform next year.

 increased reliability for all the financial metrics included across the scorecard views.

 easily reference carrier performance regarding pickup, drop-off, and POD uploads on a monthly or aggregate basis.

Has this project had the intended impact?

Based on the above metrics provide a brief summary of how this project is performing.

The project is performing well overall: it has enabled the operating teams to better track the performance of Carriers, give them feedback, and use this data to inform their booking decisions. This is reflected in the metrics above - since launch and through its two iterations, the average carrier score has increased significantly and we’ve increased trust int he Scorecard from the operation, who have begun using it to inform their booking decisions.

However, it has not fulfilled its true potential impact since we have not been able to launch the scorecard externally given pervasive completeness and correctness gaps we haven’t successfully addressed yet.

If not, what has prevented it from having the expected impact?

If possible, hypothesize what blockers are preventing this project from succeeding

No external release since there are still completeness/correctness gaps in the data (estimated events are not accurate, not tracking whether the Carrier or the operation uploads the POD) - we rely on the Carrier Reps to provide feedback to the Carriers, which is a good first step, yet the Scorecard will be more impactful once it is externally launched and carriers can monitor themselves.

Based on your analysis above, please what next steps are required for this project

Continue addressing completeness and correctness gaps in order to enable external scorecard launch

Results Readout: Carrier Scorecard at Nuvocargo

Phase 4: Results Readout - Carrier Scorecard

This phase aims to understand how well our solution is performing post-launch and what adjustments/iterations may be required.

🚦 Project Status (In Progress)

Approvals Required

Project Overview

Sign up to see more