Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

Project Zero analyzed two automated tools used by Google to uncover weaknesses in Chrome’s JavaScript engine, V8. Fuzzilli is a generative JavaScript engine fuzzer, while BigSleep employs LLM-based bug discovery. Over the past six months, BigSleep identified more vulnerabilities (45) compared to Fuzzilli (37), with a 25% overlap in reported bugs.

To assess the overlap's influence, dedicated experiments were conducted, estimating the overlap at around 60%. Nevertheless, a significant portion of vulnerabilities could be discovered by either LLMs or fuzzing alone. While LLMs' ability to find numerous vulnerabilities rapidly is promising, fuzzing and LLMs still complement each other.

The authors recommend that vendors utilize both fuzzing and agentic analysis of their code to maximize vulnerability reporting before impacting users. Fuzzing has been effective in software projects like Chrome or V8 (Chrome’s JavaScript engine) to detect bugs. Recently, the emergence of potent LLMs that can read, reason about, and code has revolutionized software development and security.

This raises questions about the comparative performance of new LLMs against traditional fuzzers. Are they replacing them, or do they complement each other? To address this, the post examines three critical questions: How many bugs does each approach discover in the real world? What is the overlap between the two approaches? What is the "shape" of bugs found by one system but not the other?

For this analysis, two systems are considered: Fuzzilli, a JavaScript engine fuzzer, and BigSleep, an agentic bug finding system. Both are in production, automatically searching for bugs in new code, making them suitable for direct comparison. Fuzzilli operates continuously across multiple instances, with new changes typically detected within a few hours.

BigSleep runs daily, reviewing new commits and generating proof-of-concept test cases for any bugs discovered. Both systems report bugs via Clusterfuzz, which performs preliminary crash triaging and submits bug reports to Chrome’s issue tracker. The analysis is confined to V8, a high-performance JavaScript and WebAssembly engine with a robust and mature testing and fuzzing framework.

Fuzzilli is a specialized fuzzing tool designed to detect bugs in such engines. Given the intricate nature of these bugs, it serves as an intriguing testbed to compare the two approaches' effectiveness in identifying deep and complex issues in well-tested software. Future work will extend this analysis to a broader range of targets, fuzzers, and agentic bug finding systems.

Next, we will explore each of the three primary questions in detail. Each section will summarize the general approach, delve into the methodology (often presented in bullet points for brevity), and discuss the results. The concluding section will attempt to interpret the findings to provide an overview of the current state of fuzzing versus AI.

While the simplest part to analyze, certain caveats must be considered. To analyze the data from Chromium’s bug tracker (where V8 bugs are recorded), the authors extracted statistics from the dedicated hotlists for each system. The analysis timeframe is limited to April to September 2026, as BigSleep’s automatic pipeline became active in April.

The raw data sources are as follows: Fuzzilli: hotlistid:5433121 created:2026-04-01..2026-09-30 BigSleep: hotlistid:8067258 created:2026-04-01..2026-09-30 The raw bug report counts are as follows: BigSleep leads in total bug reports, but this figure includes various bug types. BigSleep initially reported numerous configuration and integration issues in V8, which, while valid, have minimal real-world security implications.

To refine the comparison, vulnerabilities are filtered using the type:vulnerability: constraint, resulting in a clear advantage for fuzzers. However, this approach has limitations due to duplicates and unactionable reports (e.g., bugs closed as WAI). To address this, the authors define "actionable" bugs as those without duplicate or WAI markings and that have at least one fix commit reference.

After filtering for Type-Vuln bugs, the comparison of actionable vulnerabilities between BigSleep and Fuzzilli is more accurately depicted.