The Contract Discovery Bottleneck
AI can generate the code. We can verify the behavior. But who decides what correct means? I wrote recently about a coding agent that built me a password reset flow with a reset link that worked more than once. The bug survived because nobody had written down that a reset link should be single use. It was obvious right up until it wasn't. My argument was that as AI makes implementation cheaper,…
The contract discovery bottleneck refers to the challenge of determining what constitutes "correct" behavior when AI-generated code is used. A coding agent was used to create a password reset flow, and the resulting implementation passed tests but had a bug: the reset link could be used more than once. The reporter discovered this by adding an independently written behavioral specification, which helped the agent identify and fix the issue.
However, this process revealed that the specification itself was incomplete, as it did not account for concurrent reuse scenarios. This highlights the need for a more robust and explicit definition of correctness that can be independently verified and revised as new knowledge is discovered. The issue is not solely about who writes the tests, but rather where the definition of correct behavior comes from.
It could be generated by an AI agent, but this raises the question of where the initial specification comes from. Ultimately, the goal is to make the definition of correctness explicit enough that it can be independently inspected, challenged, tested, and revised, regardless of who is involved in the process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
