I created this template because we were having many data issues among different systems at Ramp in August of last year. There was no process so it felt chaotic, unstructured, work wasn't progressing, and problems weren't being solved. By defining a best practice and codifying a way to solve data questions, it brought structure and then the ability to move through problems quickly.

The following things trigger investigations:
- A data element drops or increases rapidly
- Unusual data or anomalies are detected either automatically or manually.

An unsuccessful investigation looks like:
- A lot of time spent investigating but didn't find a solution
- Lots of conversation but no progress
- Frustration because the altered metric then impacts performance/goals, but there isn't a clear explanation for leadership

Imagine all the sudden your grass died in your front yard. You'd start investigating. First you'd check your video camera, then the sprinklers, maybe check with neighbors. You'd build a process hoping to uncover a clue that will lead to an explanation (and then maybe a remedy). Imagine if you did all this and there was still no explanation. Not only would it be frustrating but it would be an unsolved problem. High performance companies generally aren't great at allowing for unsolved mysteries.

An example problem statement would be like this:

We've seen several cases of contacts in Hubspot that have their First Name & Last Name changed to a different name.

There are cases where the First Name & Last Name switch back and forth between the "correct" name and other name. Other cases where it switches between multiple names.

While this doesn't negatively impact systems, this behavior is very bad for our brand as customers/prospects receive emails addressed to another person.

In particular Slack is a commonly accepted workplace tool now, but many people abuse it and don't have great awareness about how running Slack convos can ultimately defeat productivity. Namely: people should strive to use Slack for realtime work - when you and your counterpart are both at your computers jamming on something. It's terrible as a "holding space" for deep, nuanced technical debugging.

At Ramp, we have an async and remote environment. This means we use Slack a lot. I was seeing investigations where 5+ people were all chatting on a Slack thread about an issue. Before you know it the Slack thread is 100+ messages long. It's impossible to follow, hard to structure, and hard to move forward. That's why in general, if you're dealing with something complex or long form, it's better to move it to a ticketing or documentation tool like Notion, Google docs, Asana, etc. That way you can have a conversation and data/ideas/next steps can be better grouped, organized, and actioned.

Teams tend to make assumptions based on past experiences. At Ramp, this meant when I joined that many teams did not trust our systems. Because of this, we were typically the first to be blamed when data didn't match expectations. People should try to explore their biases and not index towards "similar culprits" in investigations and instead open their minds to the possibility that many other things could cause an issue. This helps both with investigations and with debugging and root cause analysis.

Austin Hay

I created this template to bring structure to metrics investigations and codify best practices around solving data problems.

This is an RFC (Request for Comment) Investigation template to be used by Systems, Martech, Ops and others when requesting resources on an 

// Name of the single person responsible for this investigation. This is the person who is investigating the issue.

// Names of the people who have been assigned as being accountable to this person.

// Various parties who need to be consulted to triage the issue.

// People who need to be kept up to date about the progress of this investigation. Should include the exec sponsor for this investigation.

Once the investigation document is completed, it should be reviewed by Systems, Data, or another appropriate party for setting clear next steps.

Investigations can be rejected or requested for follow up details if they lack sufficient information.

In two to three bullet points, describe what the problem is.

In a few sentences, address the following questions:

Was there a singular event or time that the issue arose?

In this section you should attempt to provide as clear and detailed as possible a thorough understanding of the problem. Try to keep things bulleted just to the points that matter.

When discussing data elements, please describe precisely the data term and the tool you're talking about.

 "On November 3rd, something happened, causing an abrupt decrease in the Segment event Email Form Submitted, which ultimately seems to be impacting the SFDC Object, Opportunities (as reference in Snowflake, opportunities.dim table)"

In this text, you should include links to the things that you are viewing so that another investigator can come exactly to the things you are analyzing.

: tool impacted (Segment, Hubspot, SFDC, Snowflake)

Once you've laid out the owners, the problem and all the details, you should write a comprehensive list of the things that you believe may be causing the problem.

If you don't have enough systems or data knowledge to know, you should schedule time with the right systems person to come up with a thorough list of hypotheses.

Slack can be a great way to learn, but ultimately, in order to keep the investigation structured and drive towards a clear outcome, you should write your hypotheses in the investigation document.

You should try to group your hypothesis into clear structures to make it easy to add, and draw conclusions.

Last, you should use symbols, emojis or colors to denote the status of each hypothesis.

Once the root cause or issue has been identified, you can use this space to capture the findings for what drove the issue, and propose next steps.

Ideally, work to fix the issue is done rapidly once the hypothesis is proven, and then we deploy work to fix the issue. This section captures this in detail, and includes references to work, Slack notes, Github PRs or other details that highlight "what was done".

If a backfill of data is recommended for this investigation, you should file a ticket with the systems team for a data backfill.

🌟 Investigations and backfills are separate processes.

Based on what you know, you should try to describe the business impact. 

As much as possible avoid making assumptions that might weigh the worth of the investigation in one way or the other.

 Try to focus on the discrete data elements impacted and mention them explicitly.

The appendix is a great place for artifacts that may be created in the investigatory process. A simple bullet list of the items with a short sentence and a link is usually enough. Likewise, if hypotheses require their own or additional documents, you can place them here.

Data Investigation Template at Ramp

RFC - Investigation: TEMPLATE

Sign up to see more