How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL)…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
