{
  "id": 8165910,
  "title": "They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.",
  "url": "https://urgent.news/2026/09/18/they-put-7-attention-mechanisms-on-a-latin-square-then-removed-them",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-18T03:48:33.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ai_maya_063fc568e157562fd/they-put-7-attention-mechanisms-on-a-latin-square-then-removed-them-one-by-one-5ebk"
  },
  "original_language": "en",
  "account": "A paper from VIDRAFT AI Research tested the use of seven attention mechanisms arranged in a Latin square, a grid where each symbol appears once in each row and column. The key finding was that spreading these mechanisms evenly across the stack eliminated the need for architecture search to optimize layer order. Using a parameter-matched proxy with four mechanisms on a 4×4 square over sixteen layers, the researchers found that removing one mechanism at a time from the four-mechanism stack resulted in negligible effects on performance. Three of the four mechanisms could be deleted without any measurable impact, with only one mechanism showing a slight improvement. This suggests that having at least one mechanism from a different family, such as Mamba-2 or SSM, is crucial for maintaining model performance. The paper also re-ran the experiment at a larger scale, finding that penalties for removing mechanisms grew with model size, indicating that the finding is not a small-model artifact. The researchers established a pre-registered decision rule for ablation studies, requiring at least four seeds and fixing the threshold before the final run. Additionally, they included a positive control in the safety audit to ensure no causal violations in the stacked attention mechanisms.",
  "summary": "Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight identical blocks, differing only in learned weights. Nobody tested that. It is a convention, not a conclusion. A paper out of VIDRAFT AI Research ( arXiv:2609.20269 , CC BY 4.0) tests it, and the interesting part is not the headline. The headline is placement is free, composition is not . The…",
  "key_points": [
    "Seven attention mechanisms arranged in a Latin square",
    "Removing mechanisms one by one had negligible effect on performance",
    "At least one mechanism from a different family crucial for maintaining performance"
  ],
  "editors_take": "The finding that removing most attention mechanisms has little impact on performance suggests that having a diverse set of mechanisms is crucial for maintaining model performance, especially in larger models.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}