Overfitting in trading strategies: what a million trials taught me
Run enough configurations against the same history and the top row will eventually look exceptional, even if no durable edge exists. That's overfitting: mistaking the winner of a search for a discovery. I've now run 1,000,000+ machine-learning, execution and robustness trials on Bitcoin. Only two model generations ever left research for operation. The rejection rate tells you more than the…
In a study of trading strategies, researchers found that running numerous configurations on historical data could lead to "overfitting," where the best-performing model is merely a product of the search, not a genuine discovery. After conducting over one million trials on Bitcoin, only two model generations were deemed suitable for operation. The researchers emphasize that the rejection rate is a more reliable indicator than the most attractive backtest results.
One model's feature-importance profile revealed that a calendar-derived input was primarily responsible for its decision-making. However, this input was learned from the sample data rather than representing a consistent market state. When chronology intervened, the model's performance deteriorated.
To mitigate overfitting, the researchers recommend preferring stable regions over isolated peaks. They advise reproducing the exact model, inputs, and trade sequence before making any changes. Moreover, keeping the selection process and subsequent evidence separate is crucial to maintain independence. Retraining should be treated as model risk, as repeated refinements may introduce variance rather than improve adaptivity. It is also essential to audit execution separately from the signal.
The researchers highlight the difference between in-sample and out-of-sample testing. Using data from January 2018 to July 2022 for in-sample testing and July 2022 to August 2026 for out-of-sample testing, they achieved a simple return of +733.7 points, a 1:1 notional-equivalent research statistic, and a maximum drawdown of -31.04 points.
However, they refrain from publishing specific details like the horizon, barrier calibration, threshold, hyperparameters, or trained model to prevent easy replication of the production model.
Full write-ups and additional resources can be found at https://www.bitcoinai.pro/overfitting-trading-strategies/ and https://www.bitcoinai.pro/walk-forward-analysis-for-bitcoin-trading-a-complete-worked-example/. Remember, this research is not investment advice, and historical or simulated results do not guarantee future performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
