The ‘Pain in the Ass’ Math That Could Save Vibe Coding From Itself

For years, programmers have been dreaming of using math to demonstrate that their software is free from errors. This method demands engineers to craft detailed statements about the software's objectives and transform their code into formulas that can be validated. While companies typically run extensive tests to uncover as many faults in their software as possible, they rarely invest in formal verification, which can detect even more subtle issues.

A surge of startups now believes that generative AI will make formal software verification both feasible and indispensable. One such startup is &Truth, where Sun joined in August and made a public debut on Thursday. Sun's research showcased the technology's potential by leveraging AI to verify a portion of the code that encrypts conversations in the messaging app Signal.

Ilya Sergey, &Truth's chief science officer, asserts that, much like bridges and buildings do not fall apart due to math being performed before construction, software should follow the same principle. &Truth aims to verify any programming language and has secured $70 million in investments. Rival Harmonic, co-founded by Robinhood CEO Vlad Tenev, has raised $295 million and introduced Aristotle, an AI-driven verification tool supporting C and hardware designs in Verilog.

It is piloting the technology with potential users like chipmakers Tensordyne and Nvidia, among others. Recently, Anthropic and verification startup Theorem joined forces, utilizing AI to identify bugs in open-source software that previous tests missed. They also unveiled an add-on for Claude to aid other developers in verification.

The startup refrained from disclosing the initial project's costs. The main promise is a future where software operates more reliably with fewer security concerns. However, it remains uncertain if AI can extend formal verification beyond critical systems at institutions like NASA and Apple and into widespread use. The technology is still in the experimental phase.

Despite high demand, it remains unclear how much customers are willing to pay and how they intend to incorporate the technology. Formal verification assumes how computers and people interact with software, and some experts doubt its capacity to catch bugs arising from user errors and hardware failures. Will Wilson, who leads the AI software startup Antithesis, calls formal verification a "pain in the ass" for math enthusiasts like himself.

Antithesis spent millions on verifying its cybersecurity defenses, a process Wilson describes as burdensome. Yet, they still discovered additional errors using their tool, and Antithesis has also uncovered bugs in software that several of its customers had formally verified. Wilson opines, "It can provide some assurance, but only rigorous exhaustive testing can confirm your assurance is accurate."

As Vitalik Buterin, the creator of Ethereum, wrote in a recent blog post, "formal verification is not a panacea." However, proponents of the approach still think it will be a crucial safeguard against AI systems launching cyberattacks and causing chaos. Core Security components in Boeing planes, Amazon servers, and over 2.5 billion active Apple devices contain at least some formally verified software, according to company disclosures.

Typically, this code is security-related, and vulnerabilities in it could lead to severe harm if exploited. Byron Cook, a distinguished scientist and vice president at Amazon Web Services, mentions that formal verification has alleviated customer concerns about cloud security and enabled internal teams to move faster by catching major bugs early.

However, several mid-sized tech companies tell WIRED they have only formally verified one key feature of their products, if any at all. They point to a scarcity of verification engineers, costs surpassing potential risks, and most importantly, apprehensions that waiting weeks for code to be formally verified could impede innovation.

Earlier this year, Apple's security team revealed that certain issues formal verification uncovered in its "corecrypto" library for encryption could not have been detected otherwise. Nonetheless, the company emphasized the need to be cautious when incorporating new code into corecrypto to maintain a pace with reviewing it. Two cybersecurity executives from major companies that have not formally verified any code remain uncertain about whether automation will make it worthwhile.

These executives, who declined to speak publicly, suggest they might consider paying major AI labs for "verified" AI-generated code. Open-source software developers, who form the backbone of many apps, are increasingly experimenting with AI-assisted verification. The nonprofit Beneficial AI Foundation utilized a grant from Anthropic to validate encryption code this year, similar to Sun's work at Stanford.

Michael Rosenberg, a maintainer of the curve25519-dalek library, hopes for sustainable funding for ongoing verification efforts. "I have no idea how that's going to happen, though, and we certainly shouldn't rely on the goodwill of the AI companies," he states. Some startups eager to expand verification are unsure whether to deploy their tools directly to customers or through intermediaries like Anthropic.

Their general strategy is to use AI to automate much of the work involved in defining how a program should behave. For instance, specifications for a data table might require that it contain the same items before and after sorting, albeit in a different order. Further automation efforts, which might not involve AI, would convert that code into a suitable format.