For every test we run, we use a dashboard like this to make sure  each test group contains even distributions of users across various characteristics, like user segment and acquisition channel. This helps us avoid situations where test results are distorted because certain user segments are overrepresented in a test group.

This is a relatively simple query that allows us to split users into homogeneous groups and run tests that gather data that we can statistically trust.

In the first line you select your query and then you can quickly see how your data is distributed. For example, we have used this to quickly learn and see a visual representation of the channels where our clients have been acquired.

Maria Amélia Rocha

Vinicius Braz

When running tests, we use this dashboard and query to ensure that different user types are distributed evenly across test groups.

Dashboard for Monitoring Test Segmentation

Note: data in dashboard is hypothetical and for illustration only.

For every test we run we use a dashboard like this

case when ntile(x) OVER (partition by segmentation_1, segmentation_2, ...) = 1 then 'Group 1'

when ntile(x) OVER (partition by segmentation_1, segmentation_2, ...) = 2 then 'Group 2'

when ntile(x) OVER (partition by segmentation_1, segmentation_2, ...) = 3 then 'Group 3'

when ntile(x) OVER (partition by segmentation_1, segmentation_2, ...) = x then 'Group x'

-- user: unique identifier of each user (e.g. id, e-mail)

-- ntile(x): "x" is the number of groups that you want to create

-- segmentation_1, segmentation_2,...: these fields represent the variables you want to use to segment the groups homogeneously (e.g. cohort, channel)

case when ntile(2) OVER (partition by segmentation_1, segmentation_2, ...) = 1 then 'Test'

when ntile(2) OVER (partition by segmentation_1, segmentation_2, ...) = 2 then 'Control'

case when ntile(4) OVER (partition by segmentation_1, segmentation_2, ...) = 1 then 'A'

when ntile(4) OVER (partition by segmentation_1, segmentation_2, ...) = 2 then 'B'

when ntile(4) OVER (partition by segmentation_1, segmentation_2, ...) = 3 then 'C'

when ntile(4) OVER (partition by segmentation_1, segmentation_2, ...) = 4 then 'D'

Dashboard and query for separating test groups homogeneously at Cora

Dashboard for Monitoring Test Segmentation

Sample query for populating dashboard:

Sign up to see more