Software is slow because of bad design, not slow languages

People argue that AI will speed up software by rewriting it in faster languages, but this isn't necessarily true. Speeding up most programs isn't achieved by rewriting them in C or Rust. While certain languages can be significantly faster than others, such as C being 50 times quicker than Python, rewriting a Python program in C won't make it 50 times faster.

Software performance is mainly influenced by input/output operations, not the programming language itself. When a language is slow, popular libraries often exist that can optimize the code to rival C-like speed. Generally, performance improvements stem from design choices, not language selection.

Consider a straightforward CRUD website with a database. Typically, 50% of each request's time is spent querying the database to gather user data, records, and other relevant information. Translating this website into C can only accelerate the remaining 50%, which caps the possible improvements. At best, you might make requests twice as fast, though in practice, the gain is closer to 1.5x.

Most real-world programs are I/O-bound, meaning they spend most of their time waiting for data to be input or output rather than actively processing. I/O-bound programs don't just rely on databases; they may also wait for third-party APIs or disk operations. Optimizing code within curl by 100x may not be noticeable, as curl mainly waits on the network.

However, for CPU-bound work, like wikis converting markdown to HTML, rewriting in a faster language can indeed bring significant speedups, thanks to libraries like cmark-gfm that call out to faster languages.

In general, slow languages can write fast programs by leveraging such libraries. Although Python is known for its slowness, it remains a competitive option for CPU-bound tasks like machine learning and mathematics, thanks to its extensive libraries that offload these tasks to fast C code. If a Python program dedicates 80% of its time to GPU-accelerated matrix multiplications, rewriting it in C is unlikely to help, as those multiplications are already performed in optimized C or assembly code.

Not all programs heavily utilize CPU-bound work in slow languages. Many lack sufficient engineering attention, so they may have ample room for optimization. However, such programs are often unimportant; important programs with potential for optimization would have already been improved before AI made rewriting relatively easy. This doesn't mean rewriting is impossible for useful programs.

Actual Computer recently rewrote tiktoken in assembly, achieving a notable speedup. Yet, revolutionizing a field of computing with such methods is improbable. Even a fast tokenization process only contributes marginally, as inference time dominates overall program duration.

Despite these insights, most software could be faster than it currently is. I remain optimistic about the potential for significant speedups in commonly-used software, both with and without AI. However, I believe most of these improvements will arise from redesigning programs rather than rewriting them in faster languages. In 2018, Bryan Cantrill attempted to rewrite a C program in Rust due to its optimized design not meshing well with Rust's borrow checker.

Instead of rewriting, he opted for a more naive Rust implementation, which surprisingly ran 30% faster. This outcome highlights that performance is often determined by design rather than language speed. This aligns with my experience in higher-level languages, where significant speedups usually come from rewriting software to perform fewer operations.

Many strategies for enhancing software speed are unrelated to the language's performance. Often, writing in a slower higher-level language can make optimization easier, as the code's workings are more apparent. If I had rewritten the web server in assembly, identifying issues like N+1 queries or API call efficiencies would have been much harder.

Currently, many are asking AI agents to rewrite programs in Rust, C, or assembly. Although this is intriguing, it is not a serious approach to optimization. The few programs that can benefit significantly from such rewrites are limited. If your aim is to make everything faster with AI, a more productive strategy would be to use AI to profile programs, pinpoint performance bottlenecks, and suggest redesigns.

In rare cases where a hotspot is CPU-bound and cannot be refactored for efficiency, rewriting that section in a faster language may help. Nonetheless, the belief that rewriting from scratch will yield dramatic speed improvements is overstated. While there are valid reasons to rewrite software—such as portability, efficiency, and maintainability—raw speed is best achieved by focusing on the program's design.

This perspective may seem overly optimistic for those accustomed to slow languages or CPU-bound workloads but aligns closely with my experiences in interpreting these findings.