AI Now Writes Lean Proofs Faster Than Lean Can Check Them

As AI models become more adept at generating mathematical proofs, a new question arises: can we keep up with verifying and checking the work that these AI models produce? The answer is often formal verification, where the AI writes its proof as Lean code and Lean checks every step of the proof automatically. If the proof passes, it is considered correct, and this process is quick and inexpensive.

However, research conducted on OpenAI's public library of AI-generated mathematics has revealed that AI models now write Lean proofs faster than Lean can check them.

The study examined a sample of 162,170 files from the library. On average, a model could write a 450-line file in under a minute, while Lean, running on a single processor core, took 2.6 hours to check the file. Across the entire library, a model writing 1,000 tokens per second, a speed achieved by current inference services, finished first on 95% of files. As the model's speed increased, it also outpaced Lean's checking times more consistently.

The efficiency of Lean's checking process is attributed to the fact that AI models often leave the heavy lifting to the Lean software. AI-written proofs frequently contain short instructions that tell Lean to perform certain steps, such as simplifying an expression or using standard arithmetic facts. The process of finding the appropriate steps for the proof is known as "search," which consumes a significant amount of time during Lean's checking process.

The second major component is "computation," where the model generates specific examples to verify the proof, such as squaring numbers below 1,000 to check their remainders when divided by 7. This computation also takes time to execute.

The study found that storing the proof can save time during Lean's re-check process, as the complete proof is stored and can be replayed through Lean's trusted core, skipping the search step. However, the loading of modules that the file depends on still takes time. In fact, when this loading time is included, a 1,000-token-per-second model finishes first on 85 of the 100 files tested.

The computational aspects of the proof, such as squaring and remainder calculations, also remain a significant factor, as these need to be rechecked every time the proof is verified.

The findings suggest that as AI models continue to improve and write proofs at an increasingly faster pace, the process of Lean checking may become the bottleneck in trusting AI-generated mathematics. The computational components of the proofs, which are the most time-consuming part, will continue to consume large amounts of processing power.

Therefore, ensuring that the Lean checking process remains efficient and can keep up with the output of AI models will be crucial for maintaining trust in the validity of AI-written mathematical proofs.