Ancestree: unified likelihood inference of ancestral alleles under supplied or inferred genealogies

Inferring the ancestral states at each polymorphic site, distinguishing between ancestral and derived alleles, is crucial for numerous downstream population-genetic analyses. However, no existing tool can handle the full range of relevant inputs, such as plain variant data or ancestral recombination graphs (ARGs), with or without outgroups, and accommodate poly-allelic and recurrently-mutated sites.

Enter Ancestree, a new likelihood-based engine that unifies these diverse inputs within a single framework and provides full posterior distributions over the four nucleotide states at every site. The software operates in three distinct modes. The fixed-tree mode assumes a single topology across all sites and co-infers per-branch substitution rates by maximum likelihood.

The ARG mode reads a unique local tree at each site directly from a supplied ancestral recombination graph. The local-tree mode, on the other hand, samples those local trees from the genotype data using a pairwise-coalescent HMM, without requiring a pre-existing ARG. Simulated data revealed that the genealogy-based modes (ARG and local-tree) surpass accuracy and scalability, remaining robust even under outgroup configurations that defy the fixed-tree assumption.

Outgroups prove essential, as per-site inference accuracy on ingroup-polymorphic sites plummets without them, but improves significantly with a single outgroup. The most challenging sites are those fixed for the derived allele within the ingroup, lacking within-ingroup signal and requiring several deep outgroups for accurate recovery.

These same sites are highly informative, harboring the high-frequency divergence signal that underpins selection and adaptation analyses. Ancestree is now accessible on GitHub at github.com/Sendrowski/Ancestree.