They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.
Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight identical blocks, differing only in learned weights. Nobody tested that. It is a convention, not a conclusion. A paper out of VIDRAFT AI Research ( arXiv:2609.20269 , CC BY 4.0) tests it, and the interesting part is not the headline. The headline is placement is free, composition is not . The…
A paper from VIDRAFT AI Research tested the use of seven attention mechanisms arranged in a Latin square, a grid where each symbol appears once in each row and column. The key finding was that spreading these mechanisms evenly across the stack eliminated the need for architecture search to optimize layer order. Using a parameter-matched proxy with four mechanisms on a 4×4 square over sixteen layers, the researchers found that removing one mechanism at a time from the four-mechanism stack resulted in negligible effects on performance.
Three of the four mechanisms could be deleted without any measurable impact, with only one mechanism showing a slight improvement. This suggests that having at least one mechanism from a different family, such as Mamba-2 or SSM, is crucial for maintaining model performance. The paper also re-ran the experiment at a larger scale, finding that penalties for removing mechanisms grew with model size, indicating that the finding is not a small-model artifact.
The researchers established a pre-registered decision rule for ablation studies, requiring at least four seeds and fixing the threshold before the final run. Additionally, they included a positive control in the safety audit to ensure no causal violations in the stacked attention mechanisms.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
