Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude. The text spike systematically appears in early layers at a fixed initial position, independently of input content. Visual spikes vary across images, but whether their formation follows a consistent…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.