A Free Model vs 30 Security Advisory Records: An Accuracy Test You Can Rerun

A single incorrect severity classification can lead to critical package upgrades being deployed in production. An advisory classified as moderate by the model identified the package as critical, but the model's prediction was inaccurate. The reporter wanted to test the model's accuracy repeatedly, so they created a repeatable accuracy test using MonkeyCode's free model and server options.

The problem with advisory summaries is that vendors use different terminology, which makes it challenging for a model to map them to the correct action. The model does not need to be perfect, but it must be safe. The reporter scored false negatives more severely than false positives. They used 30 hand-checked test records, each representing a public advisory, to measure the model's performance.

Each record contained a description, expected package, severity, and action. The test harness was a short script that read a JSONL file, called a JSON-only completion for each record, and compared the parsed result to the expected fields. The script used the PROMPT to force JSON output with specific keys and values. The scoring function checks the predicted and expected values and returns a dictionary with boolean values for package, severity, and action.

The model had a high precision score for critical advisories but missed several moderates. The failure modes identified were vendor word mapping, package name collisions, and truncated descriptions. The reporter recommends not automating the output of this test to open merge requests, as it should only be used for triage and not as a security scanner replacement.