OpenAI gave an AI the power to block its own engineers’ code

OpenAI has equipped its AI with the ability to scrutinize and potentially halt its engineers' code during the merging process. Thibault Sottiaux, engineering lead of OpenAI's Codex team, revealed this in an interview on The Pragmatic Engineer. The automated security review system is mandatory, bypassing the need for human reviewers.

This is just one of many tasks that OpenAI is entrusting to its models, including code review, regression detection, dependency upgrades, and assisting engineers with changes that might have taken months previously.

Sottiaux mentioned that OpenAI has even benchmarked some of its code-review models as "superhuman." These models were initially developed specifically for code review but have since integrated into OpenAI's mainline models. For security purposes, a flagged issue automatically blocks the merge without any exceptions.

As AI takes over more responsibilities in reviewing code, Sottiaux believes that the human role may shift earlier in the process. OpenAI's review, deployment, and regression-catching systems are largely automated, and engineers can merge pull requests to ChatGPT, which serves around a billion active users, on the same day. The focus now seems to be on discussing the intent behind the pull request, determining whether it's the right course of action.

Sottiaux noted that engineers still need to agree on the goal and thoroughly test proposed changes. However, moving the gut-check to a pre-merge review doesn't eliminate the need for human involvement; it simply shifts it earlier in the process. Agents can handle maintenance backlogs, such as third-party library updates and routine security patches, by using clear changelogs and documentation.

This approach also applies to larger refactoring tasks, where a team might have a better architecture in mind but finds it challenging to justify spending months on a project that would take days with the help of an agent.