What Moves a LoCoMo Score? Memory Context, Answer Prompts and Judges
LoCoMo is a widely reported benchmark for long-term conversational memory, but published scores mix memory quality with choices of answer prompt, judge and baseline layout that are rarely reported together. We measure how much these choices move a score, and evaluate Becos, a team memory system for coding agents, as a black box under all of them: ten conversations (1,540 questions), two answer…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
Source: arxiv.org