Behzod Sirjani & Ravi Mehta Riff on Leveraging AI to Transform Qualitative Data into Scientific Insights 🧠💡

Leveraging AI to Transform Qualitative Data into Scientific Insights 🧠💡

Ever wondered how to make sense of the vast amount of qualitative data at your fingertips? We've got you covered! 🎧 In our latest episode of Unsolicited Feedback, we delved deep into the world of customer research in 2024 and the potential for AI to revolutionize how we handle qualitative data.

Our guest, Behzod Sirjani, shared his experience as a research consultant and advisor, helping companies make better decisions by enhancing their research and data science practices. He sees himself as a personal trainer for companies, aiding them in extracting more efficiency, results, and insights from their data-driven work. The companies are already working out (using data), but he assists them in maximizing the benefits.

📊 From Data Gatherers to Data Gardeners 🌱

Behzod’s concept here is simply that the role of a data team or a data consultant used to be to find the data for every inquiry that a product, marketing, or executive team might need. This data team would then gather the right data and put their seal of approval on it. Thanks to data warehouses like Snowflake and data visualization tools like Amplitude and Tableau, the role of the Data company shifted from being the gatekeepers of the data to being Data Gardeners - the people who organize the data, clean it up, check its validity so that other departments can reliably pull from it.

“Data scientists used to be the only ones who could go pull this metric or get this data. So they like went to the well and they like pulled up the data water and then they were like, Okay, I'll make this safe and I'll give this to Fareed and Fareed can drink now. And then we got tools like Tableau and Grafana and Looker and then data was just flowing through the pipes and the job was how do I make it safe so that anyone who turns on the tap gets the right metrics out, so they're coming to the right conclusions. ” - Behzod

The catch? This shift is rather recent and really only applies to quantitative data.

🍁 A New Supply of Qualitative Data 🍁

The utilization of tools like Grain and Gong has expanded our access to qualitative data significantly. However, this has historically been a major challenge for Data teams, as comprehending qualitative data often necessitates manual review of raw material to extract insights – a task that frequently falls on... Data teams.

⚖️ Historically There’s a Cost Mismatch Here ⚖️

The issue arises when we used to have only qualitative data on a small sample size; going through it manually felt worthwhile, even if the results aren’t scientific due to the sample size. Now, we’re collecting much more qualitative data, to the point where we could glean statistically significant results, but it's rarely worth the human time to go through that many sales calls, customer interviews, or feedback chats to get there.

🧩 Transforming Qualitative Data: The New Frontier 🚀

Ravi Mehta, our co-host, emphasized the power of LLMs in interpreting, coding, and analyzing qualitative data at scale. He shared his experience working on a system to analyze survey data, revealing how LLMs can transform qualitative data into a more quantitative form. This could be the key to unlocking the next level of data literacy and product development.

“Right now, a lot of people are looking at LLMs primarily in terms of enabling chat-based interfaces. But LLMs allow us to interpret and code and analyze qualitative data much more effectively at scale than before. I've been working on a prototype of a system to do this for survey data. And what's interesting is the combination of LLMs plus other forms of AI allow you to label at scale to cluster it into interesting topics so that you can get emergent topics of what people are talking about within the data. And then you can even feed that data into an LLM again and have it do an analysis.” - Ravi

Fareed Mosavat, took this a step further thinking about how AI could help experiment with 'vibes' in social networks. Imagine being able to experiment on the type of content encouraged on your platform and getting a quantitative outcome! 🚀

🆕 A New Data Stack Designed For “Textual” (Qualitative) Data 🆕

Fareed argues that we will hopefully see new tooling dedicated to this new frontier because there are three problems with doing this in our existing stack.

Problem 1: Structuring the Unstructured 🧩

Quantitative Data is structured in powerful tools like Snowflake, but Snowflake isn’t designed for textual (qualitative) data. From sales calls on Gong to Zoom recordings and beyond, our data landscape is sprawling and unstructured. In addition to the right Warehouse for this type of data, we’ll also need tools like Fivetran and CDPs to be able to pipe this data into a structured format.

Problem 2: Beyond Basic Alerts 🔔

Triggering alerts based on simple metrics? That's day one stuff. The real challenge lies in extracting nuanced insights from a sea of qualitative data. For instance, detecting shifts in customer sentiment post-pricing changes or identifying emerging issues before they become evergreen problems. This level of detail is crucial yet elusive, today.

The Future of Data: A Textual Stack 📊

As we delve deeper, it's clear that the future of data isn't just about tracking actions (user ID X did thing Y); it's about understanding conversations (user ID X said thing Y). This paradigm shift calls for a modern textual stack that transcends traditional quantitative analysis, blending it with the richness of qualitative insights.

We're on the cusp of something groundbreaking, and if you're building tools, technologies, or methodologies that tackle these challenges head-on, Fareed wants to chat! 💡