Scaling recipes for single-cell RNA sequencing foundation models: when do scaling laws hold?
Deep learning models exhibit empirical scaling laws whereby performance changes predictably with model size, dataset size, and training compute. Although these relationships are well established in domains such as language and image modelling, their applicability to biological data remains unclear. Here, we investigate scaling behaviour in foundation models trained on large collec tions of…
Scaling laws, which describe how performance adjusts with changes in model size, dataset size, and training compute, are well-documented in fields like language and image modeling. However, their relevance to biological data, particularly single-cell transcriptomics, is still under examination. This study explores the scaling behavior in foundation models when trained on extensive collections of single-cell transcriptomes.
The findings reveal a consistent pattern where the pre-training loss diminishes systematically with an increase in model capacity and training compute, demonstrating a power law relationship with model size. The nature and consistency of these trends vary depending on the model structure. By identifying and measuring empirical associations between the optimal learning rate and depth-to-width ratio and model size and depth (or compute), the research validates the applicability of scaling principles to transcriptomic modeling.
These insights not only confirm that scaling laws extend to the analysis of biological data but also offer a quantitative framework for predicting the benefits of additional resources and for making informed decisions about hyperparameters and architectures. This knowledge is crucial for the development of more sophisticated foundation models applicable to omics data.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
