Comparing Qini, IPW, and Doubly Robust Metrics for Uplift Modeling
Why Qini can favor the wrong uplift model under confounding, and how IPW and doubly robust evaluation change the comparison.
The article discusses the importance of evaluating causal models in uplift modeling, particularly when comparing imperfect models to the true treatment-effect rankings. It highlights a simulation where Qini, IPW, and Doubly Robust metrics were used to rank customers based on their expected benefit from marketing messages.
The article explains that Qini can sometimes assign a better score to an imperfect ranking compared to the true treatment-effect ranking. This occurs because Qini is calibrated to match the Doubly Robust (DR) metric at a reference point, which is the true causal effect ranking in the higher-selectivity, seed-42 holdout. However, even with calibration, Qini can still favor a lower-quality model due to its sensitivity to selection bias.
The article emphasizes that while having a causal model that accounts for confounding is important, the metric used to evaluate this causal model should also address selection bias. The authors delve into the evolution of their approach to measuring causal effects, starting from uplift bars, moving to the Qini metric, and finally to IPW and doubly robust scores.
They use six common learners, ranging from simple to more complex, and examine how these models perform under various degrees of confounding. The article emphasizes that understanding the true treatment effect is crucial for decision-making in marketing activities, as it helps companies allocate their limited budgets more effectively by targeting customers with the highest potential uplift.
Rewritten by urgent.news; may contain errors — check the original.
Source: hackernoon.com