My Model Had 100% Recall — Then I Realized It Was Predicting "Exoplanet" for Everything

In a recent project, a researcher aimed to build a model capable of detecting exoplanets from light curves generated by NASA's Kepler telescope. Initially, the researcher believed this to be a standard binary classification problem, so they employed a 1D CNN architecture. However, they soon encountered a major issue: the model demonstrated 100% recall but an incredibly low precision.

The model was essentially labeling every sample as an exoplanet, as it had managed to minimize loss by predicting the majority class (non-exoplanets) far too often. This predicament arose due to the severe class imbalance in the dataset—only 42 exoplanets among approximately 5,600 samples, less than 1%.

The researcher attempted various strategies to address this imbalance, such as using SMOTE to generate synthetic examples, employing SMOTETomek for oversampling and cleanup, and experimenting with threshold tuning. However, none of these techniques were truly effective because the model had scarcely encountered any exoplanet samples during its training phase.

The key to resolving this problem was to implement WeightedRandomSampler, which ensured that minority-class examples (exoplanets) were sampled far more frequently during the training process.

Another crucial modification was the use of Focal Loss, a variant of Binary Cross Entropy that down-weights easy examples and focuses the learning process on more challenging ones. This approach proved to be significantly more effective than standard Binary Cross Entropy, which treats all samples equally and often fails to learn from rare cases.

Additionally, the researcher employed a 1D Residual Network, an architecture that had previously been thought to be exclusively suited for image processing. It turned out that residual connections could also aid 1D time series data, helping to maintain signal information and stabilize gradient flow across deeper network layers.

Upon integrating these adjustments, the final model achieved an impressive F1 score of 0.9091 in the validation phase. However, this number alone was not particularly informative, as the dataset's class imbalance meant that even a single positive example could substantially sway the F1 score. To gain a more comprehensive understanding of the model's performance, the researcher conducted a Stratified 5-Fold Cross Validation, which generated F1 scores ranging from 0.4545 to 0.8421 across five different folds.

This variability underscores the importance of cross-validation in imbalanced datasets, as a single metric can offer a misleading impression of the model's accuracy.

In summary, the researcher learned that class imbalance is a far more complex issue than simply choosing an appropriate model architecture. Accuracy metrics can be highly misleading when dealing with imbalanced data, and it is essential to pay close attention to precision, recall, and F1 scores from the outset. The researcher also noted that fixing the data pipeline—such as employing weighted sampling, utilizing Focal Loss, and applying 1D Residual Networks—often proved more impactful than architectural changes.

The final report of this experiment can be accessed via the provided link. The researcher is eager to hear about similar experiences with sub-1% positive class situations on time-series data.