Feature Prioritization as an AI ML Product Manager at Amazon Alexa
Today, machine learning scientists and engineers who want to increase the rate of A/B experimentation run into issues with experiment data quality, lag in getting metrics, and lack of key features for measuring success on custom metrics. Due to space and time constraints (i.e., limitations on minimum % of customer traffic per experiment & total time to achieve statistical significance), teams such as <ABC> are only able to run A experiments per quarter even though their business requires scaling to B experiments per quarter.
However, we believe that by solving for the above pain points, we have an opportunity to increase % of experiments by D% by the end of Q4 2023. The purpose of this document is to illustrate our approach for increasing the total number of A/B experiments while driving adoption for our A/B testing platform (let’s call it Eureka) by ML scientists and engineers.
To solve this problem, we aligned on 1) goals informed by organizational priorities, 2) customer feedback to understand pain points and requirements, 3) success metrics we will track on a monthly and quarterly basis, and 4) prioritized feature roadmap to solve the pain points over the next 12 months.
Goal 1. Improve Eureka Customer Satisfaction Score (CSAT) from XX in Q4 2022 to YY by Q3 2023