A/B testing: how to test your test

Before running any A/B test, it is crucial to ensure that the testing setup itself is functioning correctly. One way to do this is by conducting an A/A test, where two identical experiences are run against each other. If there is a difference in the results, it indicates an issue with the testing setup rather than the product itself. This simple yet often overlooked validity check can reveal problems such as randomization bugs, tracking errors, or confounding variables that may have been overlooked.

Another important check is the sample ratio mismatch (SRM). If the intended split ratio (e.g., 50/50) does not match the actual ratio observed in the test results, it indicates a problem with the traffic allocation. A chi-squared goodness-of-fit test can be performed to determine whether the observed discrepancy is due to random variation or a genuine issue.

Teams typically flag SRM as a concern when the p-value is less than 0.001, as even small imbalances can become statistically significant at larger traffic volumes. For example, if a 50/50 split yields 52,000 visitors in the control group and 48,000 in the test group, the chi-squared test would yield a p-value of approximately 10^-36, signifying a highly significant imbalance.

Lastly, it is essential to examine segment-level performance in addition to the overall results. Simpson's Paradox is a phenomenon where the overall result shows one group performing better than another, but individual segments show the opposite trend. This can happen when an uneven distribution of a confounding variable occurs between the groups.

For instance, in a study comparing two kidney stone treatments, the overall success rate favored Treatment B, but when the patients were split by stone size, Treatment A performed better in both small and large stone groups. This discrepancy was due to the doctors preferentially assigning smaller stones to Treatment B, which was more suitable for that subgroup.

In an A/B test, where the assignment should be random, this pattern suggests that something correlated with the outcome was unevenly distributed between the groups. Therefore, it is wise to check the segment-level performance after a test that appears clean overall, as it may reveal underlying issues with the test setup.