How to Choose a Minimum Detectable Effect (MDE)
You have opened your sample-size calculator, entered your baseline conversion rate, kept the defaults for significance and power, and now the calculator is asking for one more number: the minimum detectable effect. It is the input that decides everything downstream — how many visitors you need, how many weeks the test runs, and whether the experiment is even worth starting. Yet it is the one…
The Minimum Detectable Effect (MDE) is a crucial input in sample-size calculators for conducting experiments. It determines the smallest true improvement that the experiment should be able to reliably detect. The MDE is not a prediction of the effect size but rather a planning input that governs the sensitivity of the experiment.
The MDE influences the sample size required for the test. The relationship between MDE and sample size is inversely proportional to the square of MDE, meaning the smaller the MDE, the larger the sample size needed. This relationship is important as it highlights the trade-off between the desired precision in detecting improvements and the amount of traffic or time required to achieve that precision.
When choosing an MDE, it's essential to consider both statistical significance and practical significance. A small MDE might require a large amount of traffic, which may not be feasible given resource constraints. Conversely, a large MDE might lead to underpowered tests that fail to detect meaningful improvements. Therefore, the MDE should be set carefully, balancing business goals with the practical limitations of the experiment.
It's also important to distinguish between relative and absolute MDE, as this distinction can greatly impact the required sample size. A 10% relative MDE translates to a different absolute MDE at different baseline conversion rates. Therefore, it's crucial to understand which convention your calculator uses and think in the units that your business actually cares about.
In summary, choosing an appropriate MDE involves considering the smallest lift worth shipping, the available traffic, and the desired sensitivity of the test. It's a business judgment that requires careful consideration of both statistical and practical factors.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
