CViT-ESP: Lightweight Pre-trained Vision Transformers for EEG-based Epileptic Seizure Prediction

Background and Objective: Accurately predicting epileptic seizures is a significant challenge that involves identifying subtle changes in brain activity preceding a seizure. While deep learning has shown potential in this field, existing models often struggle due to limited EEG datasets, high computational costs, and lack of generalizability across different patients.

This paper introduces a new approach for EEG-based seizure prediction that utilizes pre-trained Vision Transformers (ViTs) with custom modifications and optimized re-training techniques.

Methods: The key contributions of this work include: CVIT-ESP, a series of Vision Transformer architectures that replace standard patch embedding layers with custom N-dimensional CNN stages to enhance EEG representations. Additionally, ESPFormer, a lightweight transformer designed to address overfitting on small-scale EEG datasets.

To fine-tune the transformer blocks, the researchers developed a heuristic search-space reduction strategy, which effectively reduced training complexity. The methods were evaluated using the MLSPred-Bench, a patient-independent benchmark consisting of 12 diverse EEG seizure prediction tasks with varying time horizons.

Results: The results demonstrate the effectiveness of the proposed CVIT-ESP architectures over existing models. While ResNet and vanilla Transformer models achieved an AUC-ROC of 69.0%, the CVIT-ESP models achieved a maximum average AUC of 76.4%, marking the highest performance among the tested models.

Conclusions: The findings suggest that combining pre-trained ViTs with domain-specific CNN front-ends and strategic fine-tuning techniques can lead to a robust, generalizable, and resource-efficient solution for clinical seizure prediction systems. The code for the CVIT-ESP and ESPFormer models is available on GitHub at https://github.com/pcdslab/CVitEsp and https://github.com/pcdslab/ESPFormer.