Overfitting vs Underfitting: Why Your ML Model Acts Like a Student Who Either Memorizes or Doesn't Study at All

You trained a model and achieved a high score on your training data, causing you to celebrate. However, when the model encountered real-world data, it failed to perform well. This scenario illustrates overfitting, one of the two most common issues that can occur in machine learning models. The other issue is underfitting.

Overfitting and underfitting can significantly hinder the success of machine learning projects. In this article, you will learn what these problems are, how to identify each one, and most importantly, how to resolve them.

To understand the distinction between overfitting and underfitting, consider the analogy of two students preparing for a history exam. Student A memorizes every detail from the textbook, scoring 100% on a practice test with identical questions. However, when presented with a real exam containing slightly different questions, Student A struggles.

Student B glances at the chapter headings before the practice test and performs moderately, scoring 40% on both the practice and real exams. In contrast, Student C thoroughly understands the key themes and practices with various question types, achieving consistent scores of 88% on practice tests and 85% on the real exam.

In machine learning, Student A represents overfitting, as it memorizes the training data, including noise and outliers, leading to poor generalization on unseen data. Student B corresponds to underfitting, as it fails to grasp the underlying patterns in the data. Student C represents the desired balance, demonstrating appropriate learning.

Overfitting occurs when a model learns the training data too thoroughly, absorbing noise, outliers, and specific quirks unique to the training set rather than generalizable patterns. The primary sign of overfitting is exceptional performance on training data but subpar performance on test or unseen data. The model's predictions appear as a line that wildly oscillates around every data point, including random noise, indicating excessive complexity.

Overfitting can arise from limited training data, overly complex models, extended training periods, or an excessive number of features.

Underfitting, conversely, arises when a model is too simplistic to capture the true patterns within the data. The telltale sign of underfitting is poor performance on both training and test data. An underfitting model might draw a straight line through data that has a clear, non-linear pattern, missing the actual relationship entirely. Underfitting can stem from using an overly simplistic model, insufficient features, inadequate training duration, or excessive noise in the data.

To determine whether you're dealing with overfitting or underfitting, observe the following symptoms:

- Symptom: Both training and test scores are high and close (e.g., 92% and 89%). Diagnosis: Balanced model.

- Symptom: Training score is high, but test score is low (e.g., 99% and 62%). Diagnosis: Overfitting.

- Symptom: Both training and test scores are low (e.g., 55% and 52%). Diagnosis: Underfitting.

To address overfitting, consider the following strategies:

1. Obtain additional training data. Introducing diverse examples forces the model to learn genuine patterns instead of memorizing noise.

2. Simplify the model's complexity by reducing the number of layers, parameters, or employing a less intricate architecture.

3. Apply regularization techniques such as L1 (Lasso) or L2 (Ridge) regularization, which penalize the model for complexity, encouraging generalization.

4. Utilize dropout in neural networks, where randomly disabling neurons during training prevents over-reliance on specific neurons.

5. Employ early stopping, monitoring the model's performance on validation data during training, and halting the process when validation performance declines, indicating the onset of overfitting.

6. Clean and preprocess the data, eliminating noise, handling outliers, and rectifying inconsistencies, thereby reducing the amount of garbage the model may memorize.

To rectify underfitting, consider employing the following measures:

1. Boost the model's complexity by using a more advanced algorithm, such as transitioning from linear regression to polynomial regression or upgrading from a shallow neural network to a deeper one.

2. Enhance feature engineering by generating new features that better capture the underlying patterns in the data, potentially improving the model's ability to learn meaningful relationships.

3. Reduce noise within the data to make the actual signal more discernible and conducive to learning.

4. Extend the training duration to allow the model additional time to grasp the patterns in the data.

5. Reduce the level of regularization if it has been excessively applied, as this may hinder the model's capacity to learn the genuine patterns present in the data.