Are LLMs Equally Good (or Bad) at Building Secure Software?
An SCW study finds a large variance in how frontier and budget LLMs perform across different frameworks, and explains the striking cost variance.
As development teams increasingly turn to AI coding assistants to generate secure software, a crucial question emerges: do all models perform equally well—or do they vary significantly in terms of both security and cost?
An in-depth study by Secure Code Warrior and RMIT sought to answer this question by evaluating six leading large language models (LLMs) across 660 application codebases spanning eleven language and framework combinations. Each model was tested on the same tasks and subsequently analyzed by three independent SAST tools. The results, however, paint a more nuanced picture.
Firstly, no single model emerged as a universal champion when it came to security. Across OWASP's Top Ten vulnerabilities, all models scored differently, and performance varied depending on the frameworks used. GPT 5.1 excelled in Security Logging and Monitoring Failures but lagged in Insecure Design. Conversely, Gemini 2.5 Pro showed strength in Insecure Design and Identification and Authentication Failures.
Sonnet 4.5 enjoyed a balanced profile, whereas Haiku 4.5 performed moderately across most categories. In contrast, Gemini 2.5 Flash trailed in almost all areas but excelled in Server-Side Request Forgery (SSRF). GPT 5 mini, the most cost-effective option, lagged significantly, scoring below average in all categories.
While model performance varied, the costs associated with using these AI agents proved equally daunting. Token consumption, a primary cost driver, can skyrocket, particularly with agentic AI that involves numerous steps and interactions. For instance, Claude Sonnet 4.5 generated the most tokens (176 million) among the models, while GPT 5.1 and Gemini 2.5 Flash consumed fewer tokens respectively.
This token cost disparity can significantly impact IT budgets, with recent estimates indicating a 20% increase in budgets due to token usage.
In conclusion, while security performance varies among AI coding models, the rising costs of using these agents cannot be ignored. Organizations must carefully weigh both aspects—security and cost—to make informed decisions on selecting the most suitable AI coding assistant for their needs.
Written by urgent.news from DevOps.com's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
