Divide and Let the LLM Do the Rest: From Personal Experience to Architecture

Where it all began was with a simple question: just how far can we push an LLM on a complex software project? The conversation at the time seemed to be split between two extremes: AI is basically useless, albeit pretty cool; and AI is the future and programmers will eventually become obsolete. Neither of these positions resonated with the author, who believed that even with a challenging problem, getting 100% right is unlikely, but getting 0% right is also unlikely.

The goal should be to reliably measure what worked and what didn't, allowing for improvements rather than arguing about AI's effectiveness.

The author spent nearly a year trying to find that boundary. Initially, they focused on instructions, giving the model a lot of guidance and rules to dynamically select based on the task. However, this approach didn't yield the desired results. As the number of instructions increased, it became harder to manage them, and the model itself started to get confused.

The author then tried RAG (Retrieval-Augmented Generation) and atomic rules, breaking everything into small, atomic rules and having the AI retrieve the rules it needed. While this looked promising on paper, it introduced new problems: how to split the knowledge, what constituted an atomic rule, how many rules the model should retrieve, how to determine if it had enough, how to verify rules, what to do when sources became outdated, how to handle contradictions, and more.

As the project grew larger, around 5,000 lines of code, the problems became more pronounced. The difficulty wasn't just in retrieving information but in understanding the system as a whole. Perhaps the answer lay in refactoring, but the author realized that wasn't the right approach either. Instead, the author sought a way for the LLM to answer a more fundamental question: how well do I actually understand this project?

At around 3,000 lines, the context window filled up quickly, and every new task effectively became a fresh start. The model couldn't reliably maintain a mental model of the entire project, leading the author to conclude that we shouldn't try to make software equally convenient for both humans and LLMs.

The breakthrough came with the use of artifacts. By making decisions explicit and tracking what had happened during development, the author found a way to make the system more understandable. At the same time, the author switched from using OpenCode to building their own IDE, which proved to be a significant decision. From this point on, the author designed around a few simple principles: for the user, the system should not require understanding software engineering to use it; they should be able to answer questions about what happened, why it happened, what changed, and what happens next.

For software development, the system should prioritize transparency, verifiability, falsifiability, and reproducibility. The author also wanted to avoid building an enormous system just to make LLMs slightly easier to use, ensuring that the system itself remained understandable.

To illustrate the potential of this approach, consider a clinic manager who understands the clinic's operations well but may not have the skills to turn that knowledge into software. The question is how much of that work can be safely moved onto the shoulders of AI. The author doesn't argue that we should give everything to AI but rather asks which parts can be delegated.

The answer may depend less on the raw intelligence of the model than previously thought, potentially relying more on the harness around the model and choosing the right artifacts. Ultimately, there are several possible outcomes: maybe AI can't do it, AI might reliably handle only a portion of the work, or a hybrid approach might be the most effective.