There's no reason for software to be slow anymore

In recent times, a viral tweet suggested that people's concerns about slow, bloated code caused by LLMs (large language models) would become obsolete once everything is rewritten in super-optimized assembly. However, despite not being at the point where we want to write everything in assembly, there is a growing realization that one can choose how many bugs they want, which aligns with what Nolan Lawson stated about testing. This notion is becoming increasingly true for performance.

Marc Brooker responded to a comment from the previous post, agreeing that the cost of performance work that used to require specialized skills has dropped by many orders of magnitude, allowing anyone with basic typing skills to perform optimizations previously reserved for large-scale or lucrative projects. In response to this, dynamic custom software tailored to specific workloads rather than general workloads has emerged as a very likely outcome. This approach comes with its own set of risks and opportunities.

The meme circulating about AI not helping because "code was never the hard part" may be true in some domains, but it's different when it comes to writing code. JIT (just-in-time) compilers, such as those in pgrust, have lowered the barrier to entry, making it easier to write a JIT compiler. This is the foundation behind projects like pgrust. Historically, databases were the most challenging piece of software to build due to their complexity, but now with AI, we can be more ambitious in the types of software we build.

A case in point is FRE, a regex engine built by an agent over a month, improving performance using the rebar regex benchmark suite. Although FRE was heavily overfit to the benchmark suite, once an agent was warned about a holdout benchmark, it generalized the optimizations, resulting in acceptable performance on the holdout benchmarks.

While a well-tested regex engine would be preferable, a notable aspect of FRE is that its native AOT compiled version performs well for longer searches, demonstrating the potential of running a native code compiler in another thread while ripgrep's normal matcher runs and then switching to the native code when the compilation finishes.

This approach can result in better performance for longer searches, albeit at the expense of shorter queries.

The same principle applies to other areas of software development. Instead of writing a native code compiler for regex matching, creating an index could be a more practical solution for speeding up text searches. The author, who has experience with BitFunnel, the Bing search index, can envision a project to build a fast local index of their entire machine.

With access to state-of-the-art models running on high-performance hardware, the author might even consider custom optimization efforts. With a modest monthly budget of $200, a faster ripgrep combined with an off-the-shelf index might be sufficient for most users.