AI’s recursive self-improvement might not come so quickly after all
The AI industry’s boldest promise right now is that AI will soon improve itself, with almost no need for human oversight. LLMs can already write code, generate synthetic data for training, and optimize the computer chips they run on. Forecasts of explosive AI progress predict that what researchers call recursive self-improvement is on the horizon. …
A recent study has cast doubt on the idea that artificial intelligence will rapidly advance through recursive self-improvement. Led by researchers from Princeton University, the study found that AI agents can perform the necessary engineering tasks for conducting AI research but lack the creativity and judgment required for original research at top-tier academic conferences.
The study used a new method called "shadow evaluation" to test AI agents' ability to answer research questions from unpublished papers, such as whether a large language model's "personas" can be controlled by editing its weights or designing a detector for unreliable model predictions in spreadsheet data. While the agents successfully completed the engineering tasks, they failed to provide the innovative and well-reasoned research required for publication at prestigious AI conferences.
The researchers attribute this gap to the training methods used for AI models, noting that they excel at tasks with checkable answers but struggle with open-ended research due to the difficulty in creating environments to train models on such tasks. The study has limitations, as it only tested two research papers and the original authors were aware they were evaluating AI-generated work.
Written by urgent.news from MIT Technology Review's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
