Urgent.News

What's breaking now, across thousands of outlets.

Tech

We ran the control a reader asked for. It also turned out we had benchmarked the wrong model.

Two people left long, technical comments on the posts I published this week. One of them asked for an experiment I should have run before publishing anything at all. The original number was 10/10: take ten real CVE fix commits, revert each one, and ask our security agent to find what was reintroduced. It found all ten. I wrote a whole post about why that number is weak, and listed four reasons. A…

A reader requested a specific control experiment for a post published earlier this week. The original plan was to take ten real CVE fix commits, revert each one, and ask the security agent to find what was reintroduced. However, it turned out that the reviewer wasn't flagging every reverted commit. Seven out of ten ordinary reverts went unnoticed, while all ten CVE reverts were flagged.

The experiment used a control design where a reverted fix was compared to a commit with no security relevance, but the reviewer detected deletions rather than vulnerabilities. The controls showed that a reverted fix was flagged as a vulnerability, but the security framing was invented. The post discusses the three comments produced by the reviewer, which accurately described the reverted changes but promoted ordinary robustness bugs as security vulnerabilities.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Same account ID, same pet: how a deterministic buddy checker works

If two people type the same account ID into the same function, they should get the same answer every time. That's the entire spec for the checker on Claude Buddy Community Hub , a fan-made site for…

  • Same account ID yields identical pet result each time
  • Pet identity derived from unique account UUID
  • Determinism depends on consistent hash function across runtimes

A nightly-build handoff checklist for Unreal teams split between India and Europe or the US

When part of an Unreal Engine team sits in India and the rest sits in London, Austin or Toronto, the nightly build stops being a convenience and becomes the main way the two halves talk to each other.

  • Establish a cut-off time in both IST and GMT/EST, accounting for daylight saving changes.
  • Use build machine as sole source of truth for artifacts, synchronize changelist and engine revision.

More from Saturday 21 November →