Unified cell segmentation for spatial transcriptomics via transcript reassignment

High-resolution spatial transcriptomics techniques enable the measurement of gene expression as individual molecules on imaging-based platforms or as counts in subcellular bins on sequencing-based platforms. However, these methods must assign observations to cells before conducting cell-level analysis. Current segmentation approaches primarily focus on delineating cell boundaries, but transcripts that diffuse beyond their cell of origin, overlap cell boundaries, or reside in partially covered bins are often misassigned.

In this study, we introduce Tracel, a two-stage optimal transport framework designed to assign spatial observations to cells using a matched single-cell RNA-sequencing reference. Tracel begins by aligning reference cells to spatial cells to estimate the expected expression profile at each cell location. Following this, it maps transcripts to cells against these profiles. This approach is applicable to both imaging-based platforms, such as Xenium and CosMx, and bin counts from sequencing-based platforms, such as Visium HD.

Tracel has been tested across five human datasets spanning three platforms and three tissues. The results demonstrate that Tracel enhances the separation of cell types compared to existing methods. It accurately assigns cell-type markers to their correct cells, retains more cell-intrinsic transcripts, and minimizes contamination from neighboring cells.

Furthermore, Tracel preserves variation within cell types, leading to more precise differential expression analysis across spatial locations. Additionally, its expression profiles correctly attribute ligand-receptor interactions to the appropriate cell-type pairs, facilitating cell-cell communication analysis.

Tracel can be integrated into the segmentation workflow at various stages. It can either refine an existing segmentation or initiate cell segmentation starting from nucleus segmentation.