This document comprehensively analyzes an A/B test conducted on Nearpod's user activation methods. I created this to evaluate the efficacy of a new registration form introduced by Nearpod between April 19th, 2020, and May 2nd, 2020. Through this analysis, we aimed to understand if the new activation method was more effective than the existing one, and subsequently, if it should be permanently implemented. The results and insights drawn from this analysis can significantly influence user engagement and activation strategies at Nearpod.

When analyzing an A/B test, it's essential first to understand the context of why and how the test was conducted. Please always validate the assumptions behind your test, ensuring that factors like sample size and randomness are appropriate. Remember, statistically significant results should also carry practical implications. External elements, such as seasonal shifts or simultaneous marketing campaigns, influence outcomes. Using insights from one test to refine the next is beneficial, adding depth to your findings.

Be careful about common pitfalls. A low p-value doesn’t automatically imply a result's practical relevance. Resist the urge to conclude tests prematurely based on early results and know that data trends can differ when segmented. Importantly, approach your analysis without any preconceived outcomes to avoid confirmation bias. A careful and unbiased approach is the key to deriving genuine, actionable insights from your A/B testing.

Jared Husch

This document comprehensively analyzes an A/B test conducted on Nearpod's user activation methods.

A/B tests are very commonly performed by data analysts and data scientists.

For this analysis, I will be trying to understand the results of an A/B test run from April 19th 2020 to May 2nd, 2020. 

The goal is to help Nearpod understand if we should implement the new form upon registration

To get started, let's import our libraries.

import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
from scipy.stats import norm
import pymysql
%matplotlib inline
#We are setting the seed to assure you get the same answers
random.seed(42)


df = pd.read_csv('/Users/nearpoduser/Desktop/treament_control_users.csv')
df.head(10)


control_mean=df.query('group == "control"').converted.mean()

Treatment group probability of Activating

treatment_mean=df.query('group == "treatment"').converted.mean()

Lets assume that the old activation is better unless the new activation proves to be definitely better at a Type I error rate of 5%.

We will perform the sampling distribution for the difference in 

 between the two activations over 10,000 iterations of calculating an estimate from the null.

n_new=df.query('group=="treatment"').shape[0]
n_new


n_old=df.query('group=="control"').shape[0]
n_old


 values. Store all 10,000 values in a numpy array called 

# Combined conversion rate
p_combined = (convert_new + convert_old) / (n_new + n_old)

p_diffs = []

for _ in range(10000):
    new_activation_converted = np.random.binomial(n_new, p_combined)
    old_activation_converted = np.random.binomial(n_old, p_combined)
    diff = new_activation_converted/n_new - old_activation_converted/n_old
    p_diffs.append(diff)


plt.hist(p_diffs) plt.ylabel('# of Simulations') plt.xlabel('p_diffs') plt.title('Plot of 10,000 Simulated p_diffs');

# compute p value
(p_diffs>obs_diff).mean()


# plot null distribution
plt.hist(p_diffs)

# plot line for observed statistic
plt.axvline(obs_diff, c='red')


<matplotlib.lines.Line2D at 0x7fe278232820>

The value computed above is the p-value. The p-value is the probability of getting our statistic or a more extreme value if the null is true. 

, so we have evidence that our statistic was likely to not come from the null hypothesis. Therefore, we do have evidence to reject the null. We should go with the new activation!

Now we will do another AB test analysis on the same data, this time using a standard statistical test

import statsmodels.api as sm

convert_old = df.query('group == "control"').converted.sum()
convert_new = df.query('group == "treatment"').converted.sum()
n_old = df.query('group=="control"').shape[0]
n_new = df.query('group=="treatment"').shape[0]
print(convert_old)
print(convert_new)
print(n_old)
print(n_new)


 to compute your test statistic and p-value

z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new], alternative='smaller')
print(z_score, p_value)


Activation A/B Test at Nearpod

Activation A/B Test 2019

Introduction

Part I - Probability

Sign up to see more