{
  "id": 6993232,
  "title": "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases",
  "url": "https://urgent.news/2026/09/12/real-swe-benchmarking-ai-models-on-private-real-world-enterprise",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-12T20:25:48.000Z",
  "source": {
    "name": "Hacker News",
    "slug": "hacker-news",
    "url": "https://withspecific.com/benchmarks/real-swe"
  },
  "original_language": "en",
  "account": "Real-SWE is a new benchmark released by researchers to evaluate frontier AI models on private, real-world enterprise codebases. Each task in this benchmark comes from a private production codebase licensed from a real-world company, representing the actual problems software engineers face in their daily work. The benchmark focuses on two main aspects: the underlying coding artifact and the specificity of the instruction. By using native harnesses, Real-SWE reflects how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.\n\nOne example task involves fixing invoice billing so that each business charges the correct tax. Depending on the business's configuration, some may maintain their tax rate, others want the tax priced against the buyer's destination, while a customer with an exemption should not be taxed. The challenge lies in the fact that each business settles its tax differently, and an address refused by the tax authority must be reported without stopping the invoice. Pricing a destination requires interacting with the tax authority using specific addresses, product categories, and lines, all of which must be correctly documented on the issued invoice. Once an invoice is settled, the sale must be filed back to the authority under the invoice's number to ensure accurate returns reconciliation.\n\nReal-SWE tasks are inspired or lifted verbatim from private, real-world codebases, providing a more practical and challenging assessment of AI models compared to expert-generated or synthetic tasks. The researchers prioritize code written to meet an actual user or business need over code written solely for benchmarking purposes. The tasks are designed to span multiple parts of the application, requiring agents to understand existing business logic and coding patterns while keeping the surrounding system functional. Despite being slightly underspecified, the prompts are specific enough to guide the agents without omitting crucial instructions.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Hacker News Best",
        "title": "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases",
        "url": "https://urgent.news/2026/09/12/real-swe-benchmarking-ai-models-on-private-real-world-enterprise-7067773",
        "published": "2026-09-12T20:25:48.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}