Why our A/B testing tool refuses to call a winner

A/B testing tools often claim to provide clear winners, but this is not always the case. When conducting experiments, it is essential to adhere to certain guidelines to ensure accurate results. One such guideline involves the concept of a fixed-horizon test, where a sample size is committed to before starting the experiment, and a single look at the data is taken once the sample size is reached.

This approach guarantees that the probability of a false positive remains at the chosen alpha level (usually 0.05). However, repeatedly monitoring the experiment can lead to multiple looks at the data, increasing the chance of a false positive, often reaching around 20%. This phenomenon, known as repeated significance testing, occurs due to the random nature of differences between two proportions.

In a company setting, experiments are typically monitored frequently, rendering standard dashboards ineffective in providing genuine results. To address this issue, a company called ABTestly offers a tool that emphasizes the need for continuous monitoring and accurate reporting of results. When using ABTestly, three key decisions are implemented: 1.

The verdict panel indicates "Still collecting" until the decision boundary is crossed, avoiding premature conclusions. 2. The confidence interval is always displayed alongside the sample size, ensuring that lifts are interpreted correctly. 3. Sample ratio mismatch is prominently displayed, allowing for the detection of potential issues with experiment allocation.

By adhering to these guidelines, ABTestly aims to provide a more reliable and trustworthy A/B testing experience for teams using the tool.