Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.