CytoGate-Bench: an LLM benchmark for cross-panel cell gating in cytometry

Cytometry, a vital single-cell technology in clinical immunology, demands unique antibody panels and cell-type definitions for each study. Consequently, classifiers trained on one dataset cannot effectively annotate subsequent ones. Immunologists traditionally address this challenge through manual gating, sequentially partitioning populations based on multiple markers until a hierarchical structure is achieved.

To overcome this bottleneck, researchers have introduced CytoGate-Bench, a groundbreaking benchmark that transforms this iterative, step-by-step process into a zero-shot, panel-agnostic task for large language models. The benchmark consists of 23,646 expert-annotated instances, derived from 11 public flow- and mass-cytometry cohorts encompassing eight distinct marker panels.

When evaluated across six open- and closed-weight backbones, the most promising approach employs a single rectangular gate per candidate cell, matching the performance of trained, panel-specific baselines. Notably, the CytoGate-Bench formulation demonstrates greater resilience to distribution shifts compared to its counterparts.

Moreover, the benchmark reveals that navigating the hierarchical gating process stepwise yields superior results when compared to simultaneously predicting all cell types. This insight stems from analyses of data distribution shapes and curated marker priors, which prove instrumental in the performance enhancements.

Interestingly, further investigation points to the potential benefits of integrating vision capabilities and self-verification loops. These additions are found to systematically refine the gate boundaries, leading to more precise and accurate cell annotations.