Akka Tests Spec-Driven AI Delivery Across 65 Open Source Projects
Akka used 65 open-source projects to examine how specification structure, context, model selection, automated validation, and delivery guardrails affect AI assisted software porting. The experiment measured time, token use, code size, test parity, and performance, finding substantial variation across models, effort levels, and project types. By Leela Kumili
Akka utilized a spec-driven workflow to test AI-assisted software porting across 65 open source projects, evaluating various aspects such as specification structure, context, model and effort selection, automated validation, token consumption, and runtime performance. The initial phase of this experiment consumed 99.3 hours and 9.41 billion tokens, resulting in 57 out of 65 ports demonstrating improvements in lines of code or performance.
The process involved two tranches, where Akka analyzed all projects to generate specifications and implement up to 10% of each project’s surface area. Following this, 10 projects were selected for complete implementation based on specific characteristics and measurable results.
The workflow consisted of discovery, specification, porting, benchmarking, and improvement stages. During discovery, Akka analyzed code, models, schemas, and runtime behavior, while Claude, with the assistance of Akka Specify, handled implementation, testing, and review. A benchmark runner compared tests, code size, and latency across projects.
Akka discovered that structured specifications equipped with claims, evidence, and typed behavior led to more efficient first-pass implementations, despite gaps in context files related to cross-component decisions.
Subsequent areas for improvement include interface enumeration, test ingestion, provenance tracking, differential testing, and adversarial testing. In this experiment, Sonnet averaged 61 minutes per port, compared to 120 minutes for Opus, but Opus used approximately 40% fewer tokens. Higher effort settings increased token consumption without consistently enhancing efficiency.
This led to discussions among engineers regarding the factors contributing to porting efficiency, such as model capability, specification constraints, or a combination of both.
The efficiency of the porting process varied by project type; applications, frameworks, and libraries generally improved, while infrastructure and tooling often experienced median degradation. For instance, Dify saw a 143,333 times performance improvement, while Netflix Metaflow was about 100 times slower.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.