On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
