Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Real-SWE, a new benchmark released today, assesses frontier AI models on private, real-world, enterprise codebases. Each task originates from a private production codebase licensed from a real-world company. These industry-specific problems mirror the actual work their engineers perform, complete with context and complexity inherent to established products. The question arises: can an AI coding agent perform the tasks of a software engineer in the real world?

Our benchmark differs from expert-generated or synthetic tasks due to its verbatim, actual tasks engineers in real companies need to execute. We examine two aspects: the type of coding artifact and the specificity of the instructions. Both factors introduce complexities that test the limits of today's frontier models.

We employ native harnesses to mirror how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models alone. Fixing invoice billing involves ensuring each business charges the correct taxes, while exempt customers aren't taxed. Different businesses settle taxes in various ways, such as maintaining rates, pricing against buyer destinations, or charging nothing.

Pricing a destination requires collaboration with tax authorities, handling addresses, product categories, and reporting when the authority refuses. The rate, tax, and gross must appear on the issued invoice, and once settled, the sale must be filed back to the authority under the invoice's number, ensuring return reconciliation.

Invoices between European parties include both sides' VAT registrations. The authority and ledger are accessible via TAX_JAR_URL, PROD_TAX_JAR_URL, and INFLUX_URL. Each task in Real-SWE exposes only the services its workflow requires. We selected codebases through a rigorous screening process, focusing on real companies with significant usage, robust engineering teams, and demanding production workloads.

Our sample tasks stem from these codebases, including fixing invoice billing to ensure proper tax application, handling destination-based pricing, and managing tax exemptions. These tasks represent the complex, cross-functional nature of engineering changes in production environments. The work often spans multiple parts of the application, requiring agents to understand existing business logic and coding patterns while maintaining system functionality.

Interestingly, 71.4% of rollouts lasting under 10 minutes failed, compared to 73.4% for longer rollouts. Triaging multiple systems and comprehending requirements in codebases riddled with existing business logic and coding patterns pose significant challenges. Every task in Real-SWE is derived from a private, real-world codebase, making them particularly intriguing for three reasons. For access to the sample tasks, please request permission.