I Try to Kill My Ideas Before I Build Them

A major concern in today's AI-driven landscape is the ease with which one can construct the wrong solution. With just a few clicks, vast amounts of code can be generated even before it has been confirmed that the underlying idea is sound. To mitigate this risk, the author insists on testing and potentially dismissing ideas before they progress too far.

Rather than dismissing entire projects, they focus on dissecting the core claim hidden within the decision. By vocalizing the bet being made, the claim becomes tangible and testable. If the belief is that a larger model is necessary, it implies a claim that insufficient reasoning capacity is responsible for the failure. Similarly, if the author thinks a new retrieval technique will enhance memory retention, they are positing that the current method is the limiting factor.

Similarly, assuming that the standard installer process should function in a given environment is tantamount to assuming that the system behaves like a typical writable Linux setup. Upon testing, it becomes evident that the assumption was incorrect, saving valuable time and resources by preventing the development of an architectural nightmare.

On the Mycelium project, the author tried an associative-memory approach that seemed promising but ultimately performed worse than existing alternatives in all six benchmark tests. Both experiments were terminated early, preventing the creation of an architectural mess. It is crucial for a prototype to not only demonstrate that something can work but also to provide a cheap and honest opportunity for the idea to fail.

By defining what constitutes a successful outcome before conducting the experiment, rather than post hoc, the likelihood of obtaining a mediocre result that is later rationalized as encouraging is significantly reduced. Establishing clear criteria for failure is equally important. For security boundaries, it is necessary to define what absolutely must not occur.

In the context of retrieval changes, the specific improvements that must be achieved are defined. For new capabilities, a regression test is required before any level of trust can be established. Without a concrete definition of failure, the author argues that one is merely playing around without any meaningful progress. By articulating the conditions under which an idea must be terminated, the author ensures that they are not deceiving themselves and that they are focusing on the most critical assumptions.

Not all unknowns carry equal weight. While one might not be overly concerned with the appearance of a button, it is paramount to verify the functionality of the security boundary beneath it. The author emphasizes the need to prioritize the assumption that, if proven false, would invalidate the greatest amount of work. Once the critical assumption has been identified, the most cost-effective way to disprove it is to build the simplest possible test case.

By defining what constitutes a successful outcome before conducting the experiment, rather than post hoc, the likelihood of obtaining a mediocre result that is later rationalized as encouraging is significantly reduced. Ultimately, the author argues that the process is not about being pessimistic, but rather about being efficient.

With AI-driven implementation becoming increasingly cost-effective, there is less reason to be impressed by the speed at which an idea can be turned into code. Rather, the author places greater emphasis on determining whether an idea survives long enough to merit the investment of code.