I made a build profiler to understand Bun’s compile times

A developer created buildprof, an open-source tracing tool designed to visualize the performance of software builds on Linux. By recording the process tree and presenting it as a timeline, buildprof allows users to see where time is spent during the build process and identify potential areas for optimization.

The creator, inspired by a tweet about Bun's new Rust build being 5× faster on Linux than its old Zig build, wanted to explore this discrepancy. To do so, they set up scripts to replay Bun's Linux x64 CI builds on a 6-core, 12-thread Linux VM, preserving the build steps and their dependencies. The timings they obtained were comparable to those reported by the original tweet.

However, numerous factors had changed between the two measurements, so the developer turned to profiling to uncover the true causes of the build time differences. Profiling tools help uncover the sequence of events, durations, and relationships between processes involved in a build.

Build systems employ different tools, such as compilers, code generators, archivers, linkers, and arbitrary scripts, which can launch other programs. These tools often describe the build process in various ways, such as cargo crates, Ninja build edges, or CMake instructions. By tracing the subprocesses involved in a build, the developer aimed to visualize these relationships and pinpoint performance bottlenecks.

Examining a Rust build's process tree, they discovered that a single linker invocation, invoked by the linker (ld.lld), occupied over a third of the build time—over sixteen minutes in total. This single invocation accounted for roughly two-thirds of the entire build duration. The developer suspected that Full Link-Time Optimization (LTO) might be responsible for this extended period.

To confirm this suspicion, they recorded the linker's command line using buildprof, which displayed the Full LTO flag. They then enabled compiler traces to observe the internal timing events of LLD (Linker Development Library), a popular linker used in many programming languages. With the compiler traces enabled, they could clearly see that LTO was indeed consuming the majority of the linker's time.

The "OptModule" bar represented the longest segment of the linker's run, taking nearly ten minutes and encompassing several optimization passes that produce machine code.

Comparing the Zig and Rust builds, the developer found that the Rust build completed in just 2 minutes and 24 seconds, with a much shorter linker time. Moreover, the Rust linker invoked a different LTO setting (-plugin-opt=thinlto), resulting in a considerably faster build compared to the Zig build using Full LTO.

To further test the impact of LTO on the Zig build, the developer switched Bun's Zig code to use ThinLTO and recorded a new clean build alongside a fresh Full LTO build for comparison. This additional analysis revealed that changing the LTO settings can significantly reduce the link time, potentially closing the gap between the two builds.