The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation
Cooperative AI agents are evaluated against other AIs, yet human cooperation relies on implicit conventions---shared protocols for reading meaning beyond the literal message---which AI-AI benchmarks may not capture. We propose the \emph{convention gap}, the difference between the failure probability predicted from the literal content of communication and the observed failure rate, as a metric of…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
