A CRO Experiment Brief Template That Stops False Wins

A failed A/B test often stems from a vague hypothesis, choosing the metric after the test begins, or stopping the test as soon as the dashboard shows a positive outcome. To avoid these common pitfalls, a one-page experiment brief should be completed and approved before any work begins. The brief includes the following sections:

1. Observation - Describe what was seen and where, with links to evidence such as analytics funnels, heatmaps, session recordings, surveys, support tickets, and usability tests. Avoid making observations based on opinions.

2. Hypothesis - State the hypothesis using this structure: Because we observed [observation], we believe that [variable A] will cause [variable B]. We will know this when changes in [metric] occur.

3. Metrics - Define the primary metric (only one), including its definition, and specify any guards (secondary metrics) to monitor during the test. Ensure the primary metric is decided before the test launch to prevent false wins.

4. Design Variants - List the control and treatment variants, including audience and targeting details (device, market, language, traffic source), traffic split (usually 50/50), and the unit of randomization (user or session).

5. Sample size and duration - Calculate the required sample size using a calculator, based on the current baseline conversion rate and the smallest effect worth detecting for the business. Run the test for whole weeks (at least one, typically two or more full business cycles) to avoid bias from specific days of the week. If the required sample size exceeds what the page receives in a reasonable time, do not run the test.

6. Stopping rule - Document when the test will stop: when each variant reaches at least the minimum sample size, and a full two or more weeks have passed. Avoid stopping early based on early, significant results, as repeated peeking can inflate false positives.

7. QA checklist - Before launching the test, confirm the variant renders correctly on mobile and desktop in all supported languages, that there is no flicker of the original content before the variant loads, that analytics events fire identically in both variants, that the test does not break forms, checkout, or consent banners, and that exclusion rules work (internal traffic, bots).

8. Result and decision - After the test is complete, record the outcome (win, loss, or inconclusive) along with the primary metric result, guardrails, and the final decision: ship, iterate, or discard the test. Additionally, note what was learned from the test and store every concluded brief, including losses and inconclusive tests, in a searchable learning log.

Over time, this log becomes a valuable asset for the team, preventing the repeated running of similar ideas and showcasing which types of changes actually impact the audience.