Astra Held Its Ground (Truth)

TheoPhysEval has revealed a collection of 50 theoretical physics problems that were designed to be challenging for AI agents. Typically, a world expert could solve such a problem in under a day, but these issues proved to be significantly more difficult when tackled autonomously by AI without any human guidance. The best-performing AI agent was GPT-6-Astra, which was able to obtain the correct final answers with an impressive accuracy of 98% at an average cost of under $3 per problem.

One example problem from the collection involved constructing a two-parameter Hamiltonian using the suspension construction method for one-dimensional class D fermionic systems with no microscopic symmetries other than fermion parity. The task required identifying the central charge of the critical theory at the diabolical point and determining what the diabolical point destabilizes to under free-fermion perturbations.

Interestingly, the researchers initially thought Astra's score was in the high eighties, second only to another agent named Opus 5.5. However, upon closer inspection, they discovered that Astra insisted its answers were correct, even when faced with the experts' feedback. In multiple instances, the AI pointed out subtle errors in the experts' calculations that had previously gone unnoticed. As the experts revised their answers, Astra's score began to climb, ultimately resulting in a perfect score of 98% (49/50).

This surprising turn of events led the authors to conclude that the AI's answers had become the definitive ground truth for the experts, rather than the other way around. The authors expressed their amazement at this development, dubbing it "the fuck" level of performance.