Architects, Not Code Writers: Why System Design Matters More in the Age of AI

Designing systems for artificial intelligence (AI) coding agents is more critical than ever in the modern software development landscape. The emergence of AI coding agents has shifted the focus from writing code to designing systems that agents operate within. Earlier, code structure was a personal habit, but now, with AI agents continuously reading, planning, writing, and verifying throughout the codebase, system design has become a cost, speed, and correctness problem with tangible consequences.

Each AI coding task involves the agent performing several steps, all consuming tokens, which have real-world implications. Tokens represent a chunk of text that the model reads or writes, and code generally tokenizes worse than prose due to symbols, punctuation, and indentation, which add little semantic meaning per token. For instance, a function name like "calculateShippingCostForOrder" consumes roughly 8 tokens, even before the function performs any work.

The context window, where the agent holds information from previous turns, compounds the cost. With each turn, most of the context from the previous turn is sent again, leading to exponential costs. For example, a file that's 2,400 lines long, when only 90 lines are needed, results in paying for the additional 2,310 lines on every turn.

This same principle applies to edits — a monolithic approach with a single, large file consumes significantly more tokens compared to a modular approach where the bug is confined to a smaller, more manageable file. This leads to a nearly 7x cheaper solution for the same fix.

Caching can help mitigate these costs, as prompt caching allows reused context to be read back at a fraction of the cost of a fresh read. However, caching only pays off when the context is genuinely reusable turn after turn. A stable, small file, like "validators.py," is easily cacheable, while large, frequently-changing files, like a "god-file" with 2,400 lines, invalidate its cache constantly, making it more expensive in terms of context reuse.

The cost of context extends beyond just the dollar figure. The computational mechanism self-attention becomes increasingly expensive as the context size grows, leading to higher latency and GPU-hours that someone must pay for. Moreover, larger contexts not only cost more but also work worse. Research shows that accuracy drops as input length increases, even before the context window is full.

This three-fold impact—cost, compute, and accuracy—makes it clear that system design in the age of AI coding agents matters more than ever.