An Anthropic researcher just gave us a peek at self-improving AI

Anthropic's researchers have unveiled a groundbreaking study titled "Automated Researchers Can Reliably Mitigate Alignment Failures." In this research, Anthropic Fellow Chen Yueh-Han explored the potential of AI systems to self-improve and address alignment issues. By training AI models with other AI models, the system was able to improve performance on a set of 10 misaligned behaviors without compromising overall performance.

This automated method searched existing literature, proposed methods, and trained the models accordingly. Over time, effective techniques were retained, while ineffective ones were discarded, enabling rapid and scalable improvement.

The paper suggests that these results provide early evidence that automated alignment post-training could become practical in the near future. If models can improve their own alignment training, it raises the possibility that they could enhance training practices broadly, potentially rendering human AI researchers obsolete. The study acknowledges this possibility and directly compares the Automated Alignment Researcher (AAR) to human researchers, stating that the AAR outperforms human researchers on average within six hours.

The cost comparison also favors the automated approach, with the AAR costing around $4 per hour in API inference, compared to the $150 per hour spent on human researchers. However, the paper also highlights certain limitations. The automated system's effectiveness depends on the benchmarks accurately reflecting the actual alignment goals. Additionally, significant work is required to establish and maintain these benchmarks, along with the literature that the automated researchers draw from.