Accuracy Lied to Me: Precision, Recall, ROC AUC and Cross-Validation in ML Zoomcamp Module 4

In Module 3 of ML Zoomcamp, the reporter trained a lead-scoring classifier and reported an accuracy of 0.645. This number seemed sufficient at the time. However, Module 4 aimed to demonstrate the limitations of relying on accuracy alone, especially when dealing with imbalanced classes. The module provided concrete numbers from the reporter's homework notebook to illustrate these points.

The reporter first calculated the ROC AUC score for various numerical columns that could serve as features. For example, lead_score achieved an AUC of 0.788, indicating it could distinguish between leads and non-leads reasonably well. The reporter learned two key lessons: first, even a single feature like lead_score can provide meaningful separation between classes, and second, naive one-dimensional loops over numeric columns risk scoring the target against itself, leading to a perfect 1.000 AUC.

The primary model employed in Module 4 was logistic regression with a specific configuration: using one-hot encoding via DictVectorizer (fit only on the training data), a liblinear solver, and a regularization parameter (C) set to 1.0. The validation AUC score came out to 0.732, serving as a more accurate representation of the model's performance compared to the initial 0.645 accuracy.

The AUC, or Area Under the Receiver Operating Characteristic curve, is considered a valuable metric because it captures the probability that a random positive example will be ranked higher than a random negative example. This number is independent of the chosen threshold for classification and remains consistent regardless of class imbalance.

In the case of the reporter's model, the AUC score of 0.732 demonstrates its ability to rank converters above non-converters 73% of the time, without the need to choose a specific threshold.

The choice of the threshold for classification proved to be crucial. By sweeping thresholds from 0.00 to 1.00 in increments of 0.01, the reporter observed how precision and recall changed in response to this decision. The precision and recall curves intersected at a threshold of approximately 0.63, where the model achieved a precision of 0.733 and a recall of 0.725. The F1 score, which balances precision and recall, peaked earlier at 0.41 (or 0.758 when expressed as a percentage).

The default threshold of 0.5 is often used but is not inherently optimal. Instead, the reporter discovered that the optimal threshold should be determined based on the specific costs associated with false positives and false negatives in a given context. For instance, if missing a converter (false negative) carries a higher cost than sending an email to a non-converter (false positive), the threshold should be adjusted to the left of 0.5.

Conversely, if sending emails to non-converters incurs higher costs, the threshold should be adjusted to the right of 0.5. The reporter emphasized that the decision on the threshold should be guided by the specific circumstances rather than adhering to the default value.

Another critical aspect covered in Module 4 was cross-validation. The reporter performed a 5-fold cross-validation using KFold with a shuffle and a fixed random state to ensure reproducibility. The AUC scores obtained from each fold were 0.731, 0.730, 0.736, 0.741, and 0.721. The mean AUC across these folds was approximately 0.732 with a standard deviation (std) of 0.007.

This standard deviation serves as an indicator of the model's stability and generalizability to new data. It highlights the importance of reporting this metric, as it provides insight into how reliable the reported AUC score would be when applied to unseen data.

An important lesson from Module 4 was the necessity of adhering precisely to the specifications provided in the homework assignment. The reporter discovered that using a different solver (liblinear instead of the default lbfgs) led to a different standard deviation (0.007 versus 0.013) in the AUC score. This discrepancy underscores the significance of following the spec to the letter, as even minor deviations in implementation can result in different outcomes.

In this case, using the incorrect solver resulted in an inflated standard deviation and a different multiple-choice answer, emphasizing the need for careful attention to detail when executing tasks.

The reporter concluded that relying solely on accuracy to assess a model's performance can be misleading, particularly when dealing with imbalanced datasets. Module 4 demonstrated that accuracy alone does not capture the full picture. Instead, the AUC metric proved to be a more reliable indicator of a model's ability to rank positive examples higher than negative ones.

Furthermore, the reporter showcased the importance of considering the precision-recall tradeoff and selecting an appropriate threshold based on the specific costs associated with false positives and false negatives in the given context. Finally, the standard deviation from cross-validation emphasized the need for models to be stable and generalizable, as a low std value indicates that the reported AUC is likely to hold true when applied to new, unseen data.

In summary, Module 4 of ML Zoomcamp served as a powerful reminder of the limitations of relying on accuracy alone and the value of adopting comprehensive evaluation metrics like AUC, precision, recall, and cross-validation. By emphasizing the importance of considering class imbalance, threshold selection, and consistent implementation, the reporter underscored the need for a more nuanced approach to model evaluation and interpretation.