Adapters, Hypernetworks and Modular Weights
Parameter-efficient fine-tuning is a family of methods that freeze a pretrained model and train a small number of new weights alongside it. LoRA is the member everyone uses, and the reason it beat the others is a structural property rather than a quality one: its update can be added into the frozen weights and disappear. The problem the family solves Full fine-tuning of a 7B model updates all 7…
Parameter-efficient fine-tuning, represented by LoRA, is a method that freezes a pretrained model and trains just a small number of new weights alongside it. This approach offers an alternative to full fine-tuning, which updates all parameters of the model, resulting in significantly smaller per-task artefacts and a shared base model.
However, LoRA's primary advantage lies in its parallel computation, which eliminates additional sequential depth and latency costs compared to the original sequential formulation. By adjusting the rank and choosing specific modules to adapt, practitioners can control the trade-off between model capacity and computational overhead.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
