Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms

Automated cell-type annotation is essential to the majority of single-cell RNA-sequencing (scRNA-seq) analyses. However, the wide array of methods available, ranging from marker-based and correlation-based approaches to classical machine learning, deep learning, semi-supervised learning, large language models (LLM), and transformer foundation models, has outpaced thorough evaluation.

The inconsistencies in existing benchmarks, which use real datasets with uncontrollable variations in cell count, class imbalance, cell-type number, and differential-expression strength, make it difficult to attribute performance to any specific dataset property. To address this issue, researchers conducted a comprehensive benchmarking study using a Taguchi L9(34) orthogonal array that independently varied four dataset properties across seven different paradigms.

The study aimed to systematically examine the impact of these factors on annotation accuracy, while controlling for sequencing platform, database connectivity, and fine-tuned foundation models.

The findings of the study revealed some significant insights. Across all the tested paradigms, the major methods achieved comparable accuracy within a given range. The accuracy was found to be closely related to the separability of cell types in a shared expression embedding, as measured by k-nearest-neighbor (kNN) purity. This relationship was consistent across various sequencing platforms and even in fine-tuned foundation models.

This suggests that the delineation of cell types in the expression space plays a crucial role in determining the performance of annotation methods. The researchers also found that dataset structure, specifically the separability of cell types, accounted for the majority of the variance in annotation accuracy (kappa), while the choice of tool contributed only a small fraction.

This insight challenges the conventional notion that different algorithms inherently offer varying levels of accuracy.

The study also shed light on the trade-offs between computational cost and workflow accessibility. While accessible correlation-based and LLM-based approaches performed competitively, foundation models only matched their performance after undergoing fine-tuning. This implies that the accessibility and ease of use of a method should be considered alongside its accuracy.

Furthermore, the researchers emphasized that the oracle design employed in the study isolates the algorithmic capability from upstream noise, allowing for a clearer understanding of the factors influencing annotation accuracy. Based on these results, the authors argue that the near-term improvements in the field should focus on strengthening infrastructure, prioritizing tool accessibility, establishing standardized evaluation protocols, and enhancing robustness to variations in the analysis pipeline.