{
  "id": 1724331,
  "title": "Ranking Language Models by How Well They Spot Liars",
  "url": "https://urgent.news/2026/08/18/ranking-language-models-by-how-well-they-spot-liars",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-18T14:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/megapixel99/ranking-language-models-by-how-well-they-spot-liars-2mbl"
  },
  "original_language": "en",
  "account": "A new measurement method has been developed to evaluate how well language models can identify liars in a game of Mafia. The study found that ranking models based on their win rate over a batch of games is mostly due to luck, rather than the models' true capabilities. To address this issue, the researchers built a new scoring system that calculates the accuracy of each model minus the baseline probability of correctly guessing a Mafia member by chance. This new metric, called \"detection lift,\" provides a more meaningful zero and allows for more accurate comparisons between models.\n\nThe study ran 522 games across 19 different language models and found that the first two factors measured by the instrument were actually properties of the instrument itself, not the models. The leaderboard reordered itself as more games were played, revealing that the initial results were not reliable. The researchers also introduced a control group, a rule-based policy that cannot learn and has a fixed program. This control group acted as a ruler against which the performance of other models could be measured. The control's performance remained consistent across multiple batches, while the performance of the language models fluctuated significantly.\n\nThe researchers discovered that most of the turns in the game are \"day 1\" turns, where the correct answer is unobtainable because nobody has voted, contradicted themselves, or died yet. This means that the pooled detection numbers are diluted by unanswerable questions, and the dilution varies between models. Models that end their games quickly are judged mostly on their worst information turns and score worse, even if they are actually performing well. The study also found that the number of town wins varies depending on the number of seats at the table, with larger rooms having more day 1 turns and lower town win rates.\n\nOne of the most significant findings of the study is that the performance of language models differs not only in level but also in shape when looking at detection lift per day rather than pooled results. The control model starts at chance and compounds hard, while the performance of the language models shows different patterns. The rule-based control model starts at chance and improves significantly over time, while some of the language models start slightly above chance and then fluctuate around the baseline. These differences in performance patterns suggest that the complexity of the tasks and the ability to learn from the game are crucial factors in determining how well a language model can identify liars.",
  "summary": "Code: Megapixel99/social-deduction-bench A seven-player game of Mafia is a good test of whether you can tell who's lying, and a terrible one to score by who won. The Mafia can lose with perfect play because the Detective's first coin-flip investigation happened to land, and win with incoherent play because two villagers fixated on each other. Rank models by win rate over any batch you can afford…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}