This article explains how researchers separate the data a strategy was built on from the data used to confirm it, and why that split matters. It is educational, about how to evaluate any strategy or track record, and is not personalised investment advice, not a recommendation to buy, sell, or hold anything, and not a performance claim about any specific strategy. Past performance, in a backtest or live, never guarantees future results.
Any strategy can be tuned to look brilliant on the data it was built on. The only result that carries information is the one measured on data the strategy never saw while it was being designed. The distance between those two numbers is the most honest estimate you will ever get of how much of the edge was real and how much was flattery.
The short version
In-sample data is what a strategy is designed and tuned on. Out-of-sample data is held back and touched only once, to confirm. In-sample results prove almost nothing on their own, because you fit them. Out-of-sample results are the real test, and the gap between the two is a direct gauge of overfitting: a genuine edge degrades gently, an illusory one collapses.
What the two words actually mean
The vocabulary is simpler than it sounds, and getting it exactly right is half the battle. In-sample is the design data: the slice of history a researcher looks at, experiments on, and tunes against while the strategy is being built. Every choice, which indicators to use, how long a window to average over, where to set a filter, is informed by what that data shows. The strategy has, in a real sense, seen it.
Out-of-sample is the held-out data: a slice deliberately set aside at the start, never consulted during design, and used only at the end to answer one question. Does the strategy that was built on the first slice still work on the second? Because the rules were fixed before anyone looked at this data, its verdict is not something the researcher could have arranged. That is the entire source of its value.
The distinction maps onto a discipline borrowed from machine learning: the split into training, validation, and test sets. Training data is where the strategy learns its rules. Validation data is where competing versions are compared and the best is chosen. It is used repeatedly, so it slowly becomes part of what the strategy is fitted to. The test set is different in kind: it is looked at once, at the very end, and never again. The moment you use a test result to go back and change something, it stops being a test set and quietly joins the training data. Guarding that final holdout is less a statistical technique than an act of self-restraint.
Why in-sample performance is nearly worthless
The uncomfortable truth is that a spectacular in-sample record is the default, not the achievement. Give a researcher enough dials and enough history, and producing a beautiful backtest is a mechanical exercise. It tells you the person was diligent, not that the strategy is real. A model with enough freedom can be bent to pass through the specific wiggles of any sample, including the purely random ones. When you then report how it did on that same sample, you have measured your own fitting, not the market. Of course it fits. You fit it.
This is why the most flattering in-sample curves deserve the most suspicion rather than the least. A result with no ugly stretches, a ratio that strains credulity, thresholds that all landed on suspiciously tidy values, is more worrying than a good-but-scarred one, because clean perfection is exactly what tuning to noise produces. In-sample numbers set an upper bound on hope and nothing more. They describe the best case under hindsight, which is the one case you will never get to trade.
The degradation is the measurement
Here is the idea that makes the whole framework useful. You detect overfitting by comparing two numbers: how the strategy did on the data it was built with, and how the same locked rules did on the data they never saw. The drop from the first to the second is the honest estimate of how much of the apparent edge was overfitting.
- A small, graceful degradation is the signature of a real edge. Live and out-of-sample results are always a little worse than the tuned in-sample figure, some of that gap is normal luck and normal hindsight, but the strategy is recognisably the same animal. The pattern it found appears to be structural, so it survives contact with data that was not available to bias it.
- A cliff is the signature of a fit to noise. A dazzling in-sample curve followed by flat or negative out-of-sample results means the strategy captured accidents specific to the design window, and those accidents do not recur on schedule. Nothing about the beautiful backtest was load-bearing.
That gap is the primary diagnostic for the disease covered in overfitting in trading. Overfitting is the illness, the in-sample to out-of-sample degradation is the thermometer. A strategy that holds up out of sample has cleared the one bar that a merely well-tuned strategy cannot: it worked on data it could not have been secretly built to fit. Markets change, so that is never proof it will work forever.
One split is a snapshot, not the whole picture
A single in-sample or out-of-sample split has an obvious weakness: the verdict depends on which slice of history you happened to hold out. Get an unusually calm or unusually turbulent test period and the number flatters or maligns the strategy for reasons that have nothing to do with its quality. One holdout is one draw from a distribution, and one draw is a thin basis for a conclusion.
The standard answer is to roll the split forward through history, tuning on an earlier window and testing on the period immediately after, then sliding both forward and repeating, so the strategy is judged out of sample many times over many market conditions. That rolling procedure is a subject in its own right, and it is exactly what turns the one-shot concept here into a repeatable discipline, we cover it in walk-forward analysis. The concept on this page is the atom: a single honest holdout. Walk-forward is the molecule built from many of them.
The trap: a holdout you reuse is no longer a holdout
The single most important, and most violated, rule is that the test set is touched once. That once-only rule is the concept this page defends. The statistical machinery for what happens when many strategies are quietly tested against one holdout is a separate subject, which is why it lives on its own page rather than as a subsection here. This sounds trivial and is almost impossible to obey. The moment a researcher checks the out-of-sample result, dislikes it, and goes back to adjust a rule, the holdout has silently become design data. The strategy has now, indirectly, seen it: it was changed because of what that data showed. Do this a few times and the “out-of-sample” number is measuring the same thing the in-sample number did, self-fulfilment.
The subtler version is statistical rather than a single peek. Out-of-sample data is finite, and every strategy that gets tested against it uses up a little of its power to surprise you. Try enough variants against the same holdout, even honestly, one at a time, and eventually one clears the bar by chance. At that point the holdout has been quietly converted into in-sample data across the whole research program, and the winning result means far less than it appears to. Why the bar must rise with the number of attempts, and how to account for it, is the multiple testing problem. The formal tools for controlling it come from White’s Reality Check and from Harvey, Liu and Zhu on multiple testing in finance, the practical lesson is that a holdout is a consumable resource, not a renewable one.
How Shishin uses the idea
We treat the in-sample to out-of-sample distinction as the backbone of how we validate our own work, not as a chapter in a textbook. The baseline configuration that we test against history is locked: the rules are fixed and recorded, so they cannot be quietly re-tuned after we have seen how they would have done. Locking is what lets a later result count as out-of-sample at all, because there is a fixed thing being tested rather than a moving target being fitted. New ideas are held to that same discipline: a change has to survive on data it was not built on, not merely improve the historical curve.
The purest out-of-sample test we have is not any slice of the past. It is the live, paper-traded record, run forward in public. A backtest, however carefully constructed, is always a statement about history, where hindsight is available and a determined researcher can always find one more knob. The next trading day cannot be fitted, because it has not happened yet. The rules are locked, every position is visible, and the result lands where overfitting has nowhere left to hide. That is why the discipline of running exactly what was tested matters as much as the testing itself: an out-of-sample claim is only worth anything if the thing running live is the same locked thing that was validated, with no undisclosed adjustments in between. The public record lives at our track record, and the daily commitments that pin it down are at verify.
The honest limitation
Out-of-sample testing is the best tool we have, and it is still not a guarantee. Its power rests on a resource that is strictly finite: genuinely unseen data. Every test consumes a little of it, and a research program that runs long enough will, without any bad faith, gradually erode the very holdout it depends on, until yesterday’s test set has become part of today’s design data. There is no way to manufacture more untouched history. The only true source of fresh out-of-sample data is time itself, the days that have not yet happened, which is exactly why a live record that keeps accumulating, under locked rules and in the open, is worth more than any backtest, however elegant. The verdict that counts is the one you cannot arrange in advance.
Sources & further reading
- White, H. (2000). “A Reality Check for Data Snooping.” Econometrica, 68(5), 1097 to 1126.
- Harvey, C. R., Liu, Y. & Zhu, H. (2016). “… and the Cross-Section of Expected Returns.” Review of Financial Studies, 29(1), 5 to 68.
- Bailey, D. H. & López de Prado, M. (2014). “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality.” Journal of Portfolio Management, 40(5), 94 to 107.