{
  "id": 4343223,
  "title": "Your models agreed with each other. They were agreeing with themselves.",
  "url": "https://urgent.news/2026/08/30/your-models-agreed-with-each-other-they-were-agreeing-with-themselves",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-30T03:54:32.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ilya_mozerov_867dbdd91feb/your-models-agreed-with-each-other-they-were-agreeing-with-themselves-3jb0"
  },
  "original_language": "en",
  "account": "This experiment involved encoding sentences based solely on their word lengths. The sentence \"The night is long\" is represented by four clusters of lengths 3, 5, 2, and 3. The purpose of this project is to demonstrate that language models (LLMs) do not recover the intended meaning from structure and bias, instead generating projections. The researchers wanted to check if independent readings of the same encoded message would agree with each other.\n\nThey conducted two experiments, one with a known sentence and another with a different sentence. Both experiments used three arms: a treatment arm with readings of the true message, a prior control arm with readings of a different length sequence, and a random basis arm with no model at all. The agreement among the readings was measured by comparing them against random chance.\n\nIn the first experiment, the readings agreed with each other more than chance would predict, suggesting that something was getting through. However, in the second experiment, the agreement reversed. This reversal is the most useful aspect of the experiment, as it highlights two different ways a baseline can deceive you when measuring whether models agree.\n\nThe first way a baseline can lie is by inflating the agreement floor, making it rise to meet you. In this case, the random texts built from the control arm's readings were near-duplicates of each other, inflating the agreement metrics and the standard error. This led to a false conclusion that there was no signal, when in fact there was a real effect. To avoid this issue, the researchers drew the alphabet from a request independent of any message, ensuring the control arm's vocabulary could not inherit the convergence it was meant to measure.\n\nThe second way a baseline can lie is by measuring the wrong competitor. In the second experiment, the prior control arm showed higher agreement than the treatment arm, which was supposed to be the true message. This was due to the prior control arm reading a different message, resulting in higher agreement. The researchers noted that if they had only compared the treatment arm against the random basis arm, they would have concluded that independent readers converged far above chance on the encoded message, but this conclusion would have been misleading. They emphasized the importance of using multiple arms and measuring agreement against the correct competitor to avoid misinterpretation.",
  "summary": "There is a small art project in our house that encodes a sentence as nothing but its word lengths. Each word becomes a run of some symbol, repeated once per letter; the symbol itself is chosen at random and carries nothing. \"The night is long\" becomes four clusters of length 3, 5, 2, That is the entire channel. A reader — human or model — gets the lengths and nothing else. The project's README…",
  "key_points": [
    "Readings of same encoded message agreed more than chance would predict in first experiment.",
    "Second experiment showed reversed agreement, highlighting deception by baselines.",
    "Researchers emphasized importance of using correct competitor for accurate agreement measurement."
  ],
  "editors_take": "This experiment reveals that language models' apparent agreement can be misleading, as it may stem from biases in the baseline measurements rather than actual convergence on the intended meaning.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}