Dispatches from O'Reilly: The right amount of spec for agentic development

The current trend in discussions about agent-based development suggests that detailed specifications are being viewed as outdated and unnecessary overhead. Instead, the recommended approach is to provide the model with a general goal, allow it to explore, and then iterate based on the model's output. This method seems efficient on the surface, but it conceals additional costs.

While a simple prompt appears cheap and easy to implement initially, it can lead to a series of correction loops. These loops involve reviewing the model's output, clarifying the intent, requesting changes, running additional tests, identifying further gaps, and repeating the process. Ultimately, someone must determine if the result aligns with the actual goal.

This individual becomes the 'oracle.' On the other end of the spectrum, traditional formal specifications are seen as excessively costly in terms of upfront effort. Writing acceptance criteria, contract tests, or behavior-driven development (BDD) scenarios requires significant time and effort. However, the downstream cost is different, as more of the oracle becomes executable.

Tests consistently check the same condition, without getting tired, rushed, or optimistic—unlike humans who may become fatigued or make mistakes. The core question is not whether specifications are beneficial or detrimental. Instead, it revolves around determining the optimal balance of specification, examples, and executable checks that minimize the total cost.

In most cases, the ideal approach falls somewhere in the middle, incorporating sufficient structure to contain the work, concrete examples to clarify intent, and sufficient executable checks to prevent review from turning into guessing. Zero specification is not intelligent and lean; it is merely expensive and inefficient 'vibe-coding.'

Software engineering has never primarily focused on typing or producing code; it was fundamentally about deciding what should exist, what should never happen, which trade-offs matter, and defining what 'done' means once the problem intersects with the real world. Historically, teams often discovered missing specifications through human friction.

A reviewer might notice an edge case, QA could identify a path that was not adequately described, or a senior engineer might carry many of the real requirements in his head, translating them during discussions. None of these processes were elegant, but they forced ambiguity into the open. Agents change this fundamental aspect. They make implementation much cheaper and faster.

However, this also means that underspecified ideas can rapidly evolve into plausible systems before anyone has reached a complete agreement on their intended functionality. In the past, vague requirements encountered human slowness; in the agent world, vague requirements meet machine speed. This is why specification is resurging in importance.

While it was always crucial, we previously relied on implementation cost as a crude forcing function, and the outcome was treated as a process. This is the element that is frequently overlooked. People often assume the sequence is straightforward: write the specification, then let the agent implement it. However, the most crucial step—the expensive one—is often neglected.

Even a meticulously crafted specification can fail in familiar ways. It might contain contradictions or focus solely on the happy path, neglecting retries, rate limits, or partial failures. The specification could describe behavior that appears precise but cannot be reliably verified. Occasionally, the specification is precise in the wrong manner: it captures what was written, rather than what was intended.

When an agent executes a flawed specification faithfully, diagnosing the failure becomes more challenging. The implementation may appear coherent and pass the provided checks. Nevertheless, the real issue lies upstream, within the specification itself. Correcting it requires unraveling code and reasoning together. Therefore, I believe that specification validation should be treated as a separate line item.

Before beginning implementation, it is essential to ask a few fundamental questions. Is this specification internally consistent? Is it comprehensive enough for the given task? Which parts are testable? Where are we still relying on human judgment? Which failure modes are overlooked because everyone has silently assumed them? Agents can assist in this process, but their primary role should extend beyond 'writing requirements.'

A more effective prompt would be: Draft the smallest specification that would allow another agent to implement this safely. Include assumptions, nongoals, acceptance criteria, edge cases, observable outcomes, and open questions. Clearly mark which claims can be automated as tests and which still require human review. After creating the draft, hand it to a different agent, instructing it to identify contradictions, ambiguous terms, hidden dependencies, untestable claims, missing failure modes, and areas where the implementation could comply with the written criteria while still misaligning with the intended outcome.

This workflow can significantly reduce the cost of reaching a specification that is valuable for human judgment. A single agent working on a small, bounded task can often recover from loose instructions. The loop is tight, the blast radius is limited, and a human can typically guide the agent back on course if it deviates. Humans can often recognize this deviation in the first place.

However, when dealing with multi-agent systems, the situation becomes more complex. Once the output of one agent becomes the input for another, interpretive drift can compound. Agent B may not be aware that Agent A misunderstood a requirement by 10%. It simply treats the output as fact and proceeds. By the time a human examines the result, the original mistake may be obscured by several layers of seemingly competent work.

At this point, the specification is no longer merely guidance but more akin to a contract. This contract requires more than just a paragraph of intent. It demands schemas, invariants, allowed ambiguities, validation rules, and explicit failure behavior. In many instances, it also necessitates contract tests, typed interfaces, and machine-checkable handoff formats.

The handoff itself becomes an integral part of the product, which may seem less glamorous than initially anticipated but is closer to reality. This is also where BDD and executable acceptance tests play a crucial role. Their value extends beyond the methodology; they move part of the human oracle into something repeatable. When behavior is stable enough to specify precisely, an executable specification can often be cheaper than conducting an additional round of review.

However, there is another common mistake teams make when they continue to extend the specification curve, believing that more text always leads to greater safety. This is not the case, especially with current models. Chroma's work on context rot highlights the first part of this issue.