Uncovering High-Order Epistatic Interactions in GWAS via a Machine Learning-Based Feature Engineering Framework

Genome-wide association studies (GWAS) frequently overlook higher-order epistatic interactions that influence the complex inheritance of traits and diseases. Although machine learning (ML) techniques can capture nonlinear associations, interpreting these models remains difficult. A novel tree-based feature engineering framework has been introduced that utilizes Classification and Regression Trees (CART) to explicitly represent higher-order interactions as dummy variables.

Three path-based encoding strategies were examined: (i) all decision paths, (ii) leaf node paths only, and (iii) internal-node paths only. This method transforms intricate decision boundaries into discrete features that capture nonlinear interactions, which are not easily discernible through conventional association models.

The framework was tested on genetic data related to ANCA-associated vasculitis (AAV). To handle the extensive feature space generated by this encoding, various ML methods were applied to address three tasks: (1) ensemble learning using Random Forest, XGBoost, and Gradient Boosting Machine; (2) decision tree analysis employing CART; and (3) regression and classification tasks employing regularized linear regression, LASSO, support vector machine, and logistic regression.

Feature selection and regularization steps were employed to identify the most informative interaction patterns.

The findings reveal that integrating CART-derived interaction paths, especially those from high-impact regions of the tree, leads to significant enhancements in classification accuracy and model interpretability compared to utilizing the original feature space. This framework offers a robust and scalable approach to uncovering high-order genetic interactions, effectively bridging the gap between the predictive capabilities of ensemble ML methods and the need for mechanistic understanding.

By elucidating the combinatorial genetic processes underlying complex diseases, this method not only demonstrates its applicability in the context of AAV but also demonstrates its adaptability for studying the genetic architecture of diverse populations and complex traits across various populations.