I invented a CVE number to test my tool. It was real
I've been building a thing that checks whether the claims in a bug report correspond to anything that exists. Not whether the report is AI-generated — whether the file it names is in the repository, whether the function is declared, whether the version was ever tagged. To test it end to end I wrote a deliberately fabricated security report against one of my own projects. Fake file, fake function,…
A developer invented a CVE number to test their tool, which checks if claims in a bug report correspond to actual code in a repository. The fabricated report contained a fake CVE number, a fake file, a fake function, and a fake commit hash. The tool incorrectly marked the fabricated CVE as real, showing that it was focusing on whether the identifier existed rather than if it was relevant to the specific project.
This issue highlights the challenge maintainers face when dealing with bug reports and security disclosures that reference non-existent code, as they have to carefully read through each claim to identify any inaccuracies. The tool aims to address this problem by extracting checkable claims from reports, resolving them against the repository at the specified revision, and reporting any mismatches.
The developer acknowledges that the tool does not evaluate authorship or make judgments about the report's origin, and that some misses in documentation are expected and not necessarily false positives.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
