The Optimization Integrity Framework for Product Experiments

The Optimization Integrity Framework helps product teams evaluate the impact of their experiments on the overall product system. It focuses on the relationship between the primary metric, causal trade-offs, counter-metrics, and system outcomes.

The primary metric represents the local behavior the team wants to improve, such as paid conversion, engagement, churn, or transaction completion. The harder question is not what to optimize, but how the desired behavior change affects other parts of the system. Metrics improve due to changes in user behavior, and adjusting incentives, friction, defaults, timing, visibility, or access rarely affects only one behavior.

Teams often fall into the "Local Optimization Trap," where they improve a local metric while transferring part of the cost into another part of the product system. This can lead to unintended consequences that undermine the overall product performance.

The Optimization Integrity Framework consists of four key components:

1. Primary Metric: Defines the local behavior the experiment aims to change.

2. Causal Trade-Off: Describes how improvement in the primary metric could create damage elsewhere in the system.

3. Counter-Metric: A signal that reveals the damage caused by the optimization, serving as a falsification test for the causal story behind the primary metric.

4. System Outcome: Evaluates whether the broader product actually improved due to the experiment.

The framework emphasizes the importance of deriving the counter-metric from the mechanism by which success could become harmful, rather than choosing it first. By understanding the causal trade-offs, product teams can better predict and mitigate unintended consequences, ensuring that optimizations lead to overall system improvements.