A/B testing: when the standard test is the wrong tool
Last week was the standard fixed-horizon test: split traffic 50/50, wait for your pre-calculated sample size, look once, done. That's still the right default most of the time. It's not the only tool, though, and treating it as the only one gets expensive once its assumptions stop matching your situation. Why the default isn't always right A fixed-horizon test answers one question: is there a…
A/B testing, while useful, may not always be the optimal tool for determining the best course of action in certain situations. The traditional fixed-horizon test, which splits traffic evenly between two variants and waits for a predetermined sample size before drawing conclusions, may not always provide the most accurate information. This is because it assumes that the two groups being tested do not influence each other, which may not be the case in reality.
One example of this limitation was seen in a subscription business that was running a pricing test. The business, earning roughly $1 million per month, increased its conversion rate from 10% to 15% after just one week of testing. However, according to the standard test, the sample size was not yet large enough to draw any conclusions, even though the results were clearly in favor of the new pricing model.
The problem was not with the mathematics, but with the wrong question being asked. The test was designed to determine whether there was a detectable difference between the two groups at a fixed error rate, not to maximize revenue during the test.
To address this issue, multi-armed bandit algorithms offer a more dynamic approach. Rather than holding a fixed split for a fixed duration, these algorithms continuously shift more traffic towards the variant that is currently performing better, while still sending a smaller percentage of traffic to the other variants to ensure a more accurate assessment.
This exploration-exploitation tradeoff allows for a balance between learning the best option and taking advantage of the current knowledge. There are three primary strategies for implementing multi-armed bandits: epsilon-greedy, Upper Confidence Bound (UCB), and Thompson sampling. Each of these approaches has its own strengths and weaknesses, but they all aim to maximize the reward during the test.
In the pricing test example, if the pricing test had been run as a multi-armed bandit, most of the traffic would have been directed to the winning price, while a smaller slice would have continued testing the alternative. As the data came in, the split would have shifted accordingly, rather than remaining at a fixed 50/50 for five weeks.
This approach would have allowed the business to start earning the additional $500,000 in revenue sooner, rather than waiting for the fixed-horizon test to reach its predetermined sample size.
Switchback tests and hold-out tests are also alternative methods for conducting A/B tests. A switchback test alternates the whole population between the two variants on a time-block schedule, rather than splitting the population into two groups. This method is particularly useful when users are not independent of each other, as it prevents the shared supply from contaminating the results.
Hold-out tests, on the other hand, keep a slice of the population untouched by the change and compare how their behavior evolves over time. While this method can be useful for observing long-term effects, it comes at the cost of potentially missing out on a better experience for those users during the test period.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
