This article explains what overfitting is and how a disciplined research process guards against it. It is educational, about how to evaluate any strategy or track record, not personalised investment advice, and not a performance claim about any specific strategy. Past performance, in a backtest or live, never guarantees future results.
It is unsettlingly easy to build a trading strategy that looks brilliant on history and collapses the moment real money rides on it. The culprit usually has a name: overfitting, the single most common way a backtest deceives the person who built it, and the person being sold it. Understanding it is the difference between reading an equity curve as evidence and reading it as decoration.
What overfitting is
Overfitting (or curve-fitting) is when a strategy is tuned so closely to the noise and specific quirks of its historical test data that it captures patterns which will not repeat, so it looks brilliant in backtest and falls apart on new, live data. The strategy has effectively memorised the past rather than learning anything durable about how markets behave. Every random wiggle in the sample, the exact week a stock gapped, the precise month a trend stalled, gets baked into the rules as though it were a law of nature. It was an accident, and accidents do not recur on schedule.
The term comes from statistics, where a model with too many knobs can be bent to pass through every point in a dataset, including the purely random ones, tracing the scatter instead of finding the underlying relationship. In trading, the dots are historical price moves and the line is the strategy: fit it tightly enough and you explain the past flawlessly while predicting the future not at all.
The tell-tale sign: the in-sample / out-of-sample gap
Overfitting has one reliable fingerprint: a large gap between performance on the data the strategy was built with (in-sample) and performance on data it has never seen (out-of-sample), of which the live track record is the purest example. A genuine edge degrades gracefully on fresh data, a little worse, perhaps, but recognisably the same strategy. An overfit one falls off a cliff: a dazzling historical curve followed by flat or negative live results, because the noise it was tuned to does not exist outside the sample.
This is why a backtest in isolation proves so little, and why the most flattering curves deserve the most suspicion rather than the least. A result that is too clean, no ugly stretches, a Sharpe that strains credulity, parameters that all landed on suspiciously round numbers, is more worrying than a good-but-scarred one. The mechanics of how that gap is hidden, and the look-ahead errors that compound it, are the subject of why most backtests lie. This piece is about the question underneath them: why the fitting happens at all, and what an honest shop does to stop itself.
How it sneaks in
Almost nobody overfits on purpose. It creeps in through ordinary, well-intentioned research decisions, which is exactly what makes it dangerous. Four routes account for most of it.
- Too many parameters. Every threshold, filter, and tunable knob is a degree of freedom, another dial you can turn to flatter the past. A strategy with a dozen finely-set parameters has enough freedom to describe almost any history; each one added is a step away from a structural edge and toward a custom portrait of one specific sample.
- Optimising on the entire sample. Tune using all of your history, then report how it did on that same history, and you have told yourself nothing, of course it fits, you fit it. Judging a strategy on the data it was built with is the original sin, and it is astonishingly common because it feels like diligence.
- Multiple testing (data-mining bias). Run hundreds of variants, different indicators, windows, combinations, and keep only the best one, and you have almost guaranteed a great-looking result by luck alone. With enough tries, something always shines. The winner you kept is as likely to be the luckiest as the best, and its backtest tells you nothing until you account for how many lottery tickets you bought. This is why the bar for statistical significance in a real research process is set far higher than the textbook one.
- A survivorship-biased universe. Even a perfectly honest fitting process is poisoned if the data is. Test on a list of today’s surviving companies and you have quietly removed every failure, bankruptcy, and delisting before a single rule runs, a backtest that only ever traded the winners. The inflation is baked in before optimisation even begins, the subject of survivorship bias.
The first three are about fitting and the fourth about the data, but they compound: an over-tuned strategy run on a survivor-only universe produces a curve that is doubly fictional, beautiful for two independent reasons, and predictive for none.
How a serious shop detects and avoids it
No single trick makes overfitting impossible, only a set of disciplines that, applied honestly, make it far harder to fool yourself. The honesty is the load-bearing part: every defense below can be quietly defeated by a researcher who wants the strategy to work badly enough.
- Out-of-sample and walk-forward testing. Tune on one slice of history, then test, once, on a slice the strategy has never seen. Walk-forward rolls this forward repeatedly, always judging the rules on the period just after the one they learned from, which is exactly the situation they face live. The catch is the word once: peek at the out-of-sample result and go back to tweak, and that data has silently become in-sample too.
- Parameter-robustness checks. Ask whether the result survives small perturbations, nudge each threshold, shift the window by a month, try neighbouring values. A real edge sits on a broad plateau where the numbers around it all work nearly as well. An overfit one sits on a knife-edge, where the chosen value is wildly better than its neighbours, the signature of having tuned to noise.
- Correcting for the number of things tried. If you tested two hundred variants, the best one must clear a far higher bar than a single pre-registered idea would, because with that many tries, an impressive result is the expected outcome of chance, not evidence against it. Penalising the breadth of the search is what separates a discovered edge from a mined coincidence.
- Keeping the parameter count deliberately low. The most durable defense is also the most boring: prefer fewer knobs. A strategy that needs only a handful of robust parameters has less room to memorise, and what it captures is more likely to be structural. Simplicity here is not an aesthetic preference; it is a direct constraint on how badly you can overfit.
- Locking a baseline. The subtlest leak is retrospective re-tuning, quietly adjusting the strategy after you have seen how it would have done, then presenting the result as if it were the original. The defense is to lock a configuration: fix the rules, write them down, and refuse to change them after the fact without treating the change as a brand-new hypothesis on brand-new data.
How Shishin guards against it
Shishin treats overfitting as the central adversary of its research, not a footnote. Baselines are locked: the configuration that runs is fixed and recorded, so it cannot be quietly re-tuned after the results are in. New ideas are held to out-of-sample discipline and robustness checks, a change has to survive being perturbed and tested on data it was not fitted to, not merely improve the historical curve. And the system is deliberately constrained rather than knob-laden: a small number of engines, each with few parameters, is far less able to memorise a sample than one sprawling, finely-tuned model.
The part that most directly addresses the multiple-testing problem is cultural, not technical: Shishin publishes the experiments it tested and rejected. Showing only the variants that worked is precisely how data-mining bias hides, you never see the graveyard of tries that didn’t. Publishing the ideas that were measured and found wanting reveals how many lottery tickets were bought, which is the only context in which the surviving configuration means anything. That is the whole point of the experiments that failed.
And then there is the test that cannot be overfit by construction: live, paper-traded results, run forward in public. A backtest, however carefully validated against the five-year history, is always a claim about the past, where hindsight is available. The live track is the ultimate out-of-sample test, the next day is genuinely unseen, the rules are locked, and there is no way to tune to data that hasn’t happened yet. The locked system running in the open, every position visible, is the claim being tested where overfitting has nowhere to hide.
So, how do you avoid overfitting?
You cannot eliminate it, but you can corner it: use fewer parameters than you are tempted to, never judge a strategy on the data you built it with, demand that a result survive small perturbations, account honestly for how many variants you tried, lock your baseline so it cannot be quietly re-tuned, and treat live, transparent results as the verdict that outranks any backtest.
The deeper lesson is about who you trust, including yourself. An overfit strategy is rarely the work of a fraud; it is usually the work of someone who wanted the strategy to be real and let the data oblige. The defense is not cynicism but discipline, and a willingness to show the failures, not just the wins. A shop that publishes its rejected experiments and runs its locked rules live in public is making the one argument an overfit strategy never can: here is everything we tried, here is the version that survived, and here it is working on data none of us has seen. That, and not a pretty curve, is what rigour actually looks like.
Sources & further reading
- Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.” Notices of the American Mathematical Society, 61(5), 458 to 471.