Revisiting ML Model Selection With an External RealMLP Challenger

Selecting machine learning models is a crucial step in the process. The general approach involves starting with a wide range of models, narrowing down to the most promising ones, and allocating computational resources to those models. However, this method assumes that the initial selection is sufficient, which may not always be the case.

In the context of financial machine learning research, a practical example illustrates this issue. Initially, a core set of models was chosen for the ML Evaluator, which assesses trading signals. To address the risk of a narrow shortlist, an additional slot was set aside for an external challenger - a model that hadn't participated in the initial internal competition but had enough evidence to warrant a test. This model was RealMLP.

RealMLP was introduced through an external competition, highlighting the importance of considering alternative model families. This challenger was not expected to win, but it provided a valuable test of the selection process. The introduction of a challenger before the final evaluation stage helps to avoid overfitting, a common issue where the selection process starts to adapt to a specific evaluation set rather than testing the model's ability to generalize to new data.

After incorporating RealMLP, the initial model roster was re-evaluated. New models were added to the full-training roster, and RealMLP remained as an external challenger for both tracks. RealMLP's standalone performance was moderate, although it contributed to the final stacking solution. While it did not achieve outstanding results on its own, its inclusion demonstrated the importance of considering diverse model families in the selection process.