Self-Improving AI Agents บทที่ 6: Hype vs Reality + ความเสี่ยงและอนาคต

Self-improving AI agents have drawn much attention, with hype versus reality being a key topic. A recent study by Princeton researchers, led by Peter Kirgis and Sayash Kapoor, conducted a shadow evaluation where the AI agent tried to publish a NeurIPS 2026 paper. Claude Opus 4.8, the most advanced model at the time, was used for the task with a budget of $3,000, six days of computation time, and access to the internet.

The authors found that while the AI agent could solve engineering problems related to research, it lacked judgment and creativity needed to produce top conference-worthy work. This highlights the gap between hype and reality regarding AI's ability to perform self-improvement.

Another study, S3Gym, investigated whether LLMs can transition from self-testing and self-judging to self-improvement. The research revealed that while history-informed models perform well on tasks requiring specific data, they underperform on tasks needing generalizable strategies or data. Memory training can lead to negative transfer and loss of diversity, and may result in bias amplification and adversarial collapse. Memory drift, where conflicting memories accumulate, can cause agents to doubt what is true or false.

Three practical recommendations are provided to mitigate the risks associated with self-improving AI agents. Firstly, it is essential not to be blinded by hype but also not to underestimate the situation. Anthropic, the most cautious company in the field, admits that humanity is closer to recursive self-improvement than previously thought, and proactive preparation is necessary.

Secondly, the focus should be on verification rather than simply enhancing the model's capabilities. Lastly, humans must take on new responsibilities as overseers, validators, and verifiers of self-improving AI agents.