A Taxonomy-Informed Sparse DNA Foundation Model for Microbial Genomics
Microorganisms are indispensable to terrestrial ecosystems, with their genomic material underpinning critical functions and applications across agriculture, biotechnology, and human health. Although genomic language models have advanced representation learning in DNA sequences, the extensive diversity of microorganisms and imbalanced taxonomic representation in the pretraining corpora pose…
Microorganisms play a vital role in various terrestrial ecosystems, with their genomic material driving important functions and applications in agriculture, biotechnology, and human health. However, the pretraining corpora used for genomic language models often struggle to effectively model microbial genomes due to the vast diversity of microorganisms and imbalanced taxonomic representation. To address this challenge, researchers have developed MicroGlot, a taxonomy-informed microbial DNA foundation model.
MicroGlot is pretrained on an extensive dataset of 3.70 million sequences, containing a total of 378.3 billion nucleotides across 99,700 species. This extensive training data allows MicroGlot to capture the hierarchical relationships among different taxa. The model incorporates microbial taxonomic knowledge into a sparse mixture-of-experts architecture, which enables it to encode taxonomic information into the layer embeddings of the model.
Zero-shot evaluation of MicroGlot's layer embeddings reveals that the model's representations capture both phenotypic traits and taxonomic identity. Comparative analysis with a taxonomy-ablated variant, trained under the same pretraining scheme, demonstrates that incorporating taxonomic knowledge consistently improves the quality of the model's representations across all layers.
Furthermore, MicroGlot combines optimized training techniques with efficient architectural components borrowed from modern large language models. This combination results in leading zero-shot performance across layers, as well as competitive fine-tuning performance with low computational overhead. In a test set consisting of 1,000 species from major cellular domains and viral realms, MicroGlot's routing fingerprints show a greater agreement with taxonomic groups compared to tetranucleotide composition.
This indicates that MicroGlot utilizes taxonomically structured expert routing in its modeling of microbial genomes.
Overall, MicroGlot proves to be an efficient and effective DNA foundation model for microbial genomic analysis. By leveraging taxonomy-informed sparse DNA representations, MicroGlot offers a powerful tool for researchers studying microbial genomes, enabling accurate and efficient analysis across various domains within the microbial world.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
