{
  "id": 1933290,
  "title": "Are LLMs Equally Good (or Bad) at Building Secure Software?",
  "url": "https://urgent.news/2026/08/19/are-llms-equally-good-or-bad-at-building-secure-software",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-19T12:12:25.000Z",
  "source": {
    "name": "DevOps.com",
    "slug": "devops-com",
    "url": "https://devops.com/are-llms-equally-good-or-bad-at-building-secure-software/"
  },
  "original_language": "en",
  "account": "As development teams increasingly turn to AI coding assistants to generate secure software, a crucial question emerges: do all models perform equally well—or do they vary significantly in terms of both security and cost?\n\nAn in-depth study by Secure Code Warrior and RMIT sought to answer this question by evaluating six leading large language models (LLMs) across 660 application codebases spanning eleven language and framework combinations. Each model was tested on the same tasks and subsequently analyzed by three independent SAST tools. The results, however, paint a more nuanced picture.\n\nFirstly, no single model emerged as a universal champion when it came to security. Across OWASP's Top Ten vulnerabilities, all models scored differently, and performance varied depending on the frameworks used. GPT 5.1 excelled in Security Logging and Monitoring Failures but lagged in Insecure Design. Conversely, Gemini 2.5 Pro showed strength in Insecure Design and Identification and Authentication Failures. Sonnet 4.5 enjoyed a balanced profile, whereas Haiku 4.5 performed moderately across most categories. In contrast, Gemini 2.5 Flash trailed in almost all areas but excelled in Server-Side Request Forgery (SSRF). GPT 5 mini, the most cost-effective option, lagged significantly, scoring below average in all categories.\n\nWhile model performance varied, the costs associated with using these AI agents proved equally daunting. Token consumption, a primary cost driver, can skyrocket, particularly with agentic AI that involves numerous steps and interactions. For instance, Claude Sonnet 4.5 generated the most tokens (176 million) among the models, while GPT 5.1 and Gemini 2.5 Flash consumed fewer tokens respectively. This token cost disparity can significantly impact IT budgets, with recent estimates indicating a 20% increase in budgets due to token usage.\n\nIn conclusion, while security performance varies among AI coding models, the rising costs of using these agents cannot be ignored. Organizations must carefully weigh both aspects—security and cost—to make informed decisions on selecting the most suitable AI coding assistant for their needs.",
  "summary": "An SCW study finds a large variance in how frontier and budget LLMs perform across different frameworks, and explains the striking cost variance.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}