Urgent.News

What's breaking now, across hundreds of outlets.

AI

Our AI reviewer missed a terms-of-use problem. A free local panel found it in 457 seconds

This weekend we put a small tool live that turned Google's public EU ad data into a USD 39 report. Then someone finally read the data source's terms. Google's Ads Transparency Center terms say "you will not sell or monetize any part of the Ads Transparency Center, or any of the information or content within it". The paid report came down the same morning, with zero orders. The embarrassing part:…

Our AI reviewer failed to identify a critical terms-of-use issue in a newly launched tool that analyzed Google's public EU ad data. The oversight occurred because the reviewer used the same model as the tool's builder, which has a known limitation: agreement among models is not evidence of correctness. Despite the presence of prompt rules that instructed the model to only change its answer if a fundamental flaw was detected, the reviewer still missed the crucial clause in Google's Ads Transparency Center terms, which prohibits selling or monetizing any part of the data or its contents.

The AI reviewer's blind spot was shared with the tool builder, indicating a lack of diverse perspectives during the review process. To address this issue, the organization implemented several changes. They introduced critics from different labs, used open-weight models on a local GPU, and incorporated free API tiers from other providers.

They also added a mandatory first round where each critic provides their input before seeing others' opinions. Furthermore, the process now includes fixed seats with concrete checks, such as a terms lawyer, a buyer sceptic, a base-rate auditor, a red team, and an operator seat that evaluates whether a world-class operator would find the idea plausible.

To prevent similar issues in the future, the organization now requires two specific pieces of evidence before an idea can proceed: a quoted line from the data source's terms and a link showing that someone has paid for a similar service. These measures were put in place after the initial review failed, revealing that the terms seat mistakenly provided a guess instead of quoting the clause, and two other top-severity objections were incorrect.

The organization has learned that incorporating a diverse panel of reviewers, including those from different labs and with different roles, is essential to ensure a thorough evaluation process.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in AI

If you forward your calls to an AI agent, what happens to texts and voicemail?

I make Speakvora, an agent that small businesses run on their own computer, so I hear one question constantly: "I do not want to change my number.

  • Forwarding calls keeps number on original carrier, handles only calls
  • Porting number to provider moves calls and texts, takes 5-15 days
  • New business number offers quick start for calls and texts, requires carrier registration

More from Sunday 22 November →