The only feedback I got while creating this is that my “railroad tracks” emoji choice was weak, but my title made up for it.

This document was born from a data engineering roadmap brainstorming session early in the year. Our Segment setup came up as an area of improvement we wanted to focus on. At Reforge, we use Segment to track user behavior on our platform. Determining what we want to track is an inherently cross-functional process, but nobody really owned it. It led to a lot of clutter, heterogeneous naming, and confusion within Segment events, and I felt sort of embarrassed at how messy it was. We tried hiring a contractor who was an instrumentation expert to help fix our Segment process, but it didn’t work out. We realized we needed someone with both strong opinions AND the ability to get into our code base and muck around. We needed someone in-house. I realized I was best positioned to take ownership of this process, so I started by throwing out this process. This is not a groundbreaking document, but it was a great starting point!

The primary audience for this document is PMs because they have the broadest view of features across our product. In a perfect world, they’re working closely with an analyst to make sure what they want is actually measurable. But I decided to target this toward the PM instead of the analyst because they’re the ones who need this documentation the most.

The funny thing is, I’m inserting myself as a middleman into this process. Beforehand, I would get a lot of analysts coming to me after a feature had shipped, realizing they hadn’t set up the right instrumentation to measure success. I was getting all the blowback, but wasn’t involved in the process. So this is me offering an easy way to help and save time in the long run.

The most controversial opinion I embedded here is to urge people not to make a new Segment event. I could have made this a straightforward how-to, but I actually wanted to challenge the assumption that you need a new Segment event in the first place. If you aren’t going to use this data to make a decision, then you’re wasting your time. I also wanted to be very explicit about the tools you should use to make this decision, because there are many ways to shoot yourself in the foot. So I only added guidance on what I know is tried and true.

It’s REALLY important to have a consistent naming structure. It doesn’t matter as much what that structure is, just that there is one. So I needed to put a stake in the ground and say “Do it this way. Don’t ask questions.” Maybe the structure could be better and we’ll evolve it, but for now, we needed consistency above all else.

Andrew Wilson

Our instrumentation process was a mess, so I decided to take ownership of it by creating this guide as a first step. 

🛤️ So you think you need a new Segment event?

is for anyone that wants to track some user actions and thinks that our current analytics data might not be sufficient. You might be right! Maybe you’re building a new feature and want to measure its success, or maybe you’re investigating an existing feature but can’t find data for it. Either way, this guide is for you!

These are the roles that will need to be involved to complete the process end to end:

Decide which user actions you want to measure.

Determine if we have sufficient existing analytics to measure the user actions.

Collaborate with an analyst or data engineer to define a new Segment event. 

Assign an engineer/data engineer to implement the new Segment event.

Confirm that the new Segment event is working as expected

Actually use the new Segment event to make decisions.

1. Decide which user actions you want to measure

Unless you are actually going to use the user action data to make decisions, you’re about to waste yours and everyone else involved’s time. 

So if you don’t yet know which decision the data is going to help you make, now’s the time to sort that out.

be explicit about which user actions you want to measure

be able to demo in the product exactly which user actions you want to track

it’s not critical to write anything down at this point. What matters is that 

2. Determine if we have sufficient existing analytics to measure the user actions.

go to Amplitude and look to see if there are already Segment events firing for the user actions you want to measure. Look yourself up in User Look-up to see events firing in near-real time while you do the actions. Details here

Maybe the page view events are sufficient?

Maybe adding properties to an existing event is sufficient?

Maybe we have existing analyses in Metabase that are sufficient?

The first two questions you should be able to answer yourself, but Metabase search kind of sucks so don’t hesitate to ask an analyst or data engineer for help.

3. Collaborate with an analyst or data engineer to define a new Segment event.

The desired outcome of this step is adding the new event and its properties to the Production Tracking Plan

If you give a data engineer a seat at the table, they might be able to handle the actual implementation for you in the next step, which is a plus! 

We use Segment’s recommended Object Action Framework for naming events. Think 

Be pragmatic about which properties to include in the event, you don’t need to boil the ocean and new properties can easily be added later. Focus on what you need to know to actually make decisions

Always include a user_id for logged-in users

4. Assign an engineer/data engineer to implement the new Segment event.

Make a Jira ticket with a link to the new event you defined in the Production Tracking Plan and an explicit description of which user action(s) should trigger it. A loom of you actually doing the user actions would be a nice-to-have.

Go through your normal channels for eng support, or reach out to #pod-platform to ask a friendly data engineer to implement it.

5. Confirm that the new Segment event is working as expected

Whoever implemented the event probably tested it locally before making their PR. But you should confirm that it’s working as expected before basing decisions on it.

I recommend using Amplitude to test event implementation. Assuming the PR has been deployed to production, basically do the same thing you did in Step 2 and make sure the events are flowing and the properties are correct.

6. Actually use the new Segment event to make decisions.

Your friendly neighborhood analyst or data engineer can help you leverage the newly-created tracking event to aid in your decision making.

Segment Event Guide at Reforge

🛤️ So you think you need a new Segment event?

Outline

Sign up to see more