JIT Compiling Code in 5μs

In the realm of programming, fast Just-In-Time (JIT) compilation has often been considered a complex endeavor, requiring expertise in assembly language. However, recent advancements in artificial intelligence have made it much simpler to create JIT compilers that compile code at runtime with remarkable speed. This breakthrough is particularly impactful for databases, which previously relied on slower compilation methods like LLVM or C/C++ code generation.

One noteworthy example of this advancement is the pgrust project, where the author initially doubted the feasibility of implementing a JIT compiler. Yet, with the help of AI, the process turned out to be far more straightforward than anticipated, contributing to the project's overall performance. The pgrust JIT compiler can compile code in just 5 microseconds, allowing it to JIT compile every SQL query, rather than just a fraction of them.

JIT compilation is a technique that generates executable code at runtime, often resulting in significant performance improvements, typically ranging from 2 to 5 times faster, and sometimes even more. This technique is especially valuable when programming language interpreters encounter runtime information that significantly alters program behavior. JIT compilers are not limited to programming languages; they also play a crucial role in parsing data, especially when the schema is only known during runtime.

To illustrate the power of JIT compilation, the author created a toy regular expression engine that supports only literal strings and repetition (i.e., the regex *). This engine was implemented using Rust structures, enabling it to handle strings such as "b(an)*" without the need for alternation or lookbehind features.

A straightforward interpreter for this regular expression engine was developed, consisting of under 20 lines of code. However, when benchmarked against a manually written version optimized for the regex, the interpreter proved to be 10-20 times slower. This stark contrast highlighted the potential for performance improvement through JIT compilation.

To leverage JIT compilation, two main steps are required. First, the assembly code for the desired runtime operations must be generated. This is achieved through a process called copy-and-patch. In this approach, predefined assembly templates, known as "stencils," are customized based on the specifics of the operation being JIT compiled. By combining these customized stencils, a program can be constructed at runtime that achieves performance comparable to manually written code.

The author demonstrates this process by generating ARM64 assembly code for the regex "b(an)*". The code is broken down into distinct sections, starting with the prologue, which initializes the program's state by setting the stack pointers. Following the prologue, the code checks for the literal string "b" and handles any non-matching characters by jumping to a fallback logic block. Subsequently, the repetition section (an)* is addressed, which involves backtracking logic to handle potential failed matches.

The final section of the generated assembly code concludes the regex by checking if the end of the string has been reached. If so, the function returns 1 to indicate a successful match; otherwise, it signifies a failed match. This JIT-compiled regular expression engine demonstrates the immense potential for performance gains when employing JIT compilation techniques.