JIT Compiling Code in 5μs

Fast JIT compilation was once considered a black art. Historically, only a deep understanding of assembly language was required to create a fast JIT compiler. However, advancements in artificial intelligence have made it much simpler to write a JIT compiler with swift compile times. This principle applies not only to databases but also to various other fields, such as regular expression engines.

Thomas, the author of pgrust, initially doubted the feasibility of implementing a JIT compiler. Yet, with the aid of AI, the task proved to be much easier than anticipated. Consequently, pgrust's JIT compiler compiles code in approximately 5 microseconds, allowing for the JIT compilation of every SQL query, as opposed to just a subset of queries.

To build a fast JIT compiler, one must understand the process of generating assembly code at runtime. This technique, known as JIT compilation or "Just In Time" compilation, can produce significant performance improvements, often 2-5 times faster, and sometimes even more. JIT compilation is particularly useful when runtime information significantly alters a program's behavior, such as in programming language interpreters or data parsing situations where the schema may not be known until runtime.

For this demonstration, an example of a toy regular expression engine using JIT compilation will be implemented. The engine supports only two features: literal strings and repetition (i.e., the regex '*'). The regular expression will be pre-parsed into Rust structures, eliminating the need for a separate parser. The engine will be able to support strings like "but no alternation or lookbehind or anything like that."

The regular expression engine consists of three types of nodes: a literal string node, a repetition node, and a concatenation node, which combines two nodes. The engine's interpreter is relatively straightforward, as it is only around 20 lines of code. However, when benchmarked against handwritten code specifically designed for the regex "b(an)*", the interpreter proved to be 10-20 times slower.

To improve performance, the author employed JIT compilation. The process involves generating assembly for the desired code, packaging it into a function, and calling it like any other code in the program. The author uses a variant of the copy-and-patch approach for generating assembly code. This approach involves using templates or "stencils" for different operations, modifying them based on specific operation requirements, and combining them to create a program at runtime with performance similar to handwritten code.

The path to creating a general regular expression engine with JIT compilation involves several steps. First, the ARM64 assembly code for "b(an)*" will be examined. Then, repeated instruction sequences will be converted into reusable stencils. An emitter will be written to fill and combine these stencils from the regex abstract syntax tree (AST). Finally, the generated instructions will be copied into executable memory so that Rust can call them like a normal function.