One AI Answer, Eight Brands: Designing a Benchmark Without Multiplying the Evidence

A recent AI benchmark involving eight luxury jewelry brands revealed a surprising discrepancy in visibility metrics. Piaget appeared in all four answers regarding brands with verifiable official China channels but was absent from recommendations for wedding jewelry. This highlighted the need for multiple metrics to accurately assess AI visibility, rather than relying on a single metric.

The benchmark collected 12 valid raw answers, which were then analyzed separately from the raw answers themselves. Key judgments, such as mentions, recommendations, and channel accuracy, were derived from the answers and linked to specific brands. Invalid responses were treated as attempts rather than negative results, and the dataset retained audit trails for failed attempts.

The benchmark's findings demonstrated that different buyer decisions, such as wedding shortlist recommendations, daigou-risk answers, and official-channel assertions, required distinct metrics. An aggregate AI visibility score would have masked these differences, leading to a misleading diagnosis. By maintaining separate denoters and numerators for each metric, the benchmark offered a more nuanced understanding of AI visibility for each brand and use case.