UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning
Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the…
UPLiFT, or Uncertainty Parameterized Activation for Lightweight Fine-Tuning, presents an innovative approach to adapting pretrained protein language models (PLMs) that challenges the traditional method of adjusting network weights. In contrast to these weight-only adaptations, UPLiFT formulates the adaptation process as an optimization task within the parameterized activation space.
This novel approach leverages the work on curvature-tuning to enhance adaptation capacity, even when the underlying model, whether pretrained or randomly initialized, remains unchanged.
The key feature of UPLiFT lies in its learnable non-linearity, which facilitates the modeling of uncertainty over the activation space. This adaptability is achieved through fine-tuning within a variational inference framework, making UPLiFT particularly useful in scenarios where parameter efficiency is crucial. Evaluations on various protein datasets demonstrate that UPLiFT achieves competitive performance while utilizing 3-11 times fewer trainable parameters than traditional weight-only adaptations.
Adding a Bayesian twist to UPLiFT further refines the model by improving calibration without significantly compromising the predictive performance of deterministically fine-tuned models. This aspect is especially valuable in residue-level tasks, where UPLiFT demonstrates a narrow performance margin compared to a low-rank adaptation baseline. This indicates the potential of UPLiFT as an efficient and effective fine-tuning technique tailored for PLM-based predictors in the protein modeling domain.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
