Cross-validation, and the split that hides a model's failure
Cross-validation is how most models get their score. Six points worked through by hand show how k-fold works, and how the same model scores 0.80 or −0.67 depending only on how the data was split. A model has to be scored on data it was not trained on, or the score only measures memory. Holding back one test set works, but with little data the score then depends heavily on which rows landed in it.…
Cross-validation is a common method used to assess the performance of a model. By splitting the data into equal parts, or "folds," the model's performance can be evaluated by training on k-1 folds and testing on the remaining fold. This process is repeated k times, ensuring that each fold serves as the test set once. However, there is a hidden flaw in this method that can lead to misleading results.
In the provided example, a straight-line model fitted to data with two groups is scored using k-fold cross-validation with three folds. While the model achieves a decent score of 0.80, the same model trained on one group and tested on another group yields a significantly lower score of -0.67. This discrepancy arises because the model is only tested on data it has already seen during training, leading to an inflated score.
Furthermore, the example demonstrates how the choice of folds can heavily influence the model's performance score. Splitting the data by group rather than randomly mixing the data leads to a much worse score, indicating that the model has learned to fit each group rather than the overall relationship. This highlights the importance of choosing folds that reflect the real-world scenarios in which the model will be used.
In the author's research on predicting the stiffness of metal plates, a similar issue was encountered. A neural network model performed exceptionally well when trained on round-hole cells but scored below zero when tested on other hole families. Splitting the data based on the group (hole type) rather than individual rows revealed that the model had simply memorized the patterns within each group rather than learning the underlying relationship between hole type and stiffness.
To mitigate this issue, the author suggests using group-based cross-validation techniques, such as GroupKFold and LeaveOneGroupOut, which ensure that the model is tested on data from different groups than those used for training. This approach helps prevent the model from being overly optimistic about its performance and encourages it to learn more generalizable patterns.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
