This article explains a statistical problem in strategy research and the standard ways of correcting for it. It is educational and general, about how to read any backtest or track record, not personalised investment advice, and not a recommendation to buy, sell, or hold anything. Past performance, backtested or live, never guarantees future results.
Show us one dazzling backtest and we will show you nothing: we cannot tell whether it is a discovery or the luckiest draw from a thousand tries. The multiple-testing problem, also called data-snooping, is that a single impressive result is close to meaningless unless you also know how many strategies were tested to find it. Get that count wrong and your significance test is not measuring skill. It is measuring the breadth of your search.
The short version
If you test enough strategies, some will look excellent by pure chance. A significance threshold built for a single, pre-registered idea is far too lenient once you have tried dozens or hundreds. The fix is to raise the bar in proportion to the number of trials, using family-wise error control, false-discovery-rate control, or a return haircut. Any honest backtest verdict has to be adjusted for how hard you looked.
Why searching manufactures false positives
Start with one honest experiment. You pick a rule in advance, run it once, and ask how unlikely the result would be if the rule had no edge at all. The conventional 5% bar (a t-statistic near 1.96) means: if there were truly no edge, a result this good would appear by luck only one time in twenty. For a single pre-registered test, that is a reasonable line.
Now run the same test on twenty unrelated, edgeless rules. Each has a one-in-twenty chance of clearing the bar by luck. So on average one of them will, and it will look every bit as convincing as a real discovery. Nothing was learned, yet you now hold a “significant” strategy. This is the whole trap in one sentence: the 5% error rate is per test, and strategy research runs an enormous number of tests. The probability that at least one edgeless rule clears the bar climbs fast: about 64% after twenty independent tries, over 99% after a hundred. With enough tickets, a winner is arithmetic.
The insidious part is that the search is usually invisible, even to the person who did it. Every abandoned indicator, every window length nudged, every filter added and removed, every stop tightened is another trial, whether or not anyone wrote it down. The published strategy is the survivor of a selection process, and a survivor tells you little unless you can see the graveyard it climbed out of. This is the deep engine behind overfitting: the quiet accumulation of attempts that makes the best of them look inevitable.
Two ways to be wrong: family-wise error and false discovery
Statisticians formalise “too many tries” two ways, and the difference decides which correction is appropriate.
- Family-wise error rate (FWER). The probability of making even one false discovery across the whole family of tests. Controlling it is strict: you are demanding that the entire batch be almost certainly free of a single fluke. Appropriate when one false positive is costly, for example when a fluke strategy would actually be funded and traded.
- False-discovery rate (FDR). The expected proportion of your declared discoveries that are false. This tolerates a controlled fraction of mistakes in exchange for finding more of the real effects. Appropriate for wide screens where you expect several genuine edges hiding among many nulls and can afford to re-test the shortlist. Introduced by Benjamini and Hochberg (1995), it is the workhorse of modern large-scale testing.
Neither is “correct” in the abstract. FWER asks “are we sure nothing here is a fluke?” FDR asks “of the things we are excited about, what share are likely flukes?” A serious research process picks the one that matches the cost of being wrong, and states which it used.
The corrections, from blunt to sharp
Once you accept that the bar must rise with the number of trials, the question is only how to raise it. There is a family of tools, trading simplicity against power. They also sit on the two sides of the split above: Bonferroni and White’s Reality Check control the family-wise error (no fluke anywhere), while Benjamini-Hochberg controls the false-discovery rate (a capped fraction of flukes among the winners).
- Bonferroni. The blunt instrument: to control the family-wise error at 5% across N tests, require each individual test to clear 5% / N. Test a hundred rules and the survivor must clear roughly a 0.05% bar, a much higher t-statistic than 1.96. It is dead simple and famously conservative: because it assumes the worst about how the tests relate, it can reject real edges alongside the flukes, especially when the strategies are correlated (as trading variants almost always are).
- False-discovery-rate control. The Benjamini-Hochberg procedure ranks all the p-values and applies a graded threshold, so it keeps far more genuine discoveries than Bonferroni while still capping the fraction of duds. In finance, Harvey, Liu and Zhu (2016) apply exactly this logic to the “factor zoo,” the hundreds of published return predictors, and conclude that after honestly accounting for the multiple testing, most newly-proposed factors should be treated as false. Their headline recommendation is stark: a newly-claimed edge should clear a hurdle closer to a t-statistic of 3.0 than the textbook 1.96.
- Return (Sharpe) haircuts. Rather than a pass/fail bar, you can shrink the reported performance itself to reflect the search. The deflated Sharpe ratio (Bailey and López de Prado) asks what the best of N tried strategies would score by luck alone, given the number of trials and the non-normality of returns, and discounts your observed Sharpe by that expected maximum. It is the Sharpe-specific member of this family, and it earns its own treatment in the deflated Sharpe ratio.
- Reality Check and the SPA test. White’s Reality Check (2000) and Hansen’s Superior Predictive Ability test (2005) go a step further: instead of correcting a single strategy, they ask whether the best strategy from an entire searched universe beats a benchmark by more than a data-snooping adjustment would predict. They build the null distribution of the best-of-N result by bootstrapping the actual returns, typically with the stationary bootstrap of Politis and Romano (1994), which resamples blocks so the resampled series keeps the autocorrelation and fat tails of the original. That last detail matters, because financial returns violate the tidy independence the simple formulas assume.
Why a t-stat near 2 is not enough
The intuition worth carrying away is the “expected best of N.” Draw N strategies that are all pure noise and keep the top performer. Its apparent Sharpe or t-statistic is the maximum of N random draws, and the maximum of many draws drifts steadily upward as N grows: for independent pure-noise tests, the expected largest t-statistic scales roughly with the square root of 2 ln N, so a hundred edgeless tries produce a best-of expected t near 3.0, comfortably past the textbook 1.96. Test enough noise and the winner clears a t of 2 almost by default, purely because you selected the extreme. The number that looked like skill is just the right tail of a distribution you sampled hard.
So a lone t-statistic of 2 is uninterpretable without its denominator: two out of how many tries? The same 2 is strong evidence for a single pre-registered rule and no evidence at all for the winner of a five-hundred-strategy sweep. This is why the honest reading of any result depends on a number almost nobody reports: the size of the search that produced it. It is also the precise reason a real research process sets its bar for statistical significance well above the textbook one.
How Shishin holds itself to the adjusted bar
We treat the multiple-testing problem as the default condition of our own research, not a hazard that happens to other people. Everything we run has been selected, and selection inflates. So the system is judged against a multiple-testing-adjusted bar rather than the lenient textbook one, and we test significance distribution-free (a block bootstrap that preserves the fat tails and clustering of the real return stream) rather than leaning on a parametric t that assumes away exactly the features that make markets hard. We do not restate the specific threshold here; the bar we chose, and why, lives in the significance piece.
The part that most directly attacks data-snooping is disclosure. The single hardest input to any correction is the trial count, and the only way to make it credible is to show the search. So we publish the ideas we tested and the data rejected: the sector caps, the conviction sizing, the drawdown throttles, the entry variants. That graveyard is what turns the surviving configuration from an unverifiable boast into a countable result, and it is the whole point of the experiments that failed. Then there is the test no amount of searching can rig: the four engines run forward, locked, in public, where tomorrow is genuinely unseen and there is nothing left to mine. Every fill and NAV tick sits on the live track, and the daily commit-and-reveal log at verify makes the record impossible to edit after the fact.
The honest limitation: you can never count the trials perfectly
Every correction above needs one number as input, the number of things tried, and that number is fundamentally unknowable in practice. Do you count only the strategies you formally backtested, or also the ones you rejected on a glance at a chart? The prior work of everyone who has ever mined the same market history, whose survivors seeded your own priors? The variants you did not run because a colleague already tried them? There is no clean answer. The effective search that shaped any strategy is always larger than the tally you can produce.
The consequence is humility about direction. Because the true trial count is almost certainly higher than the one you can justify, every multiple-testing adjustment is a floor on the correction, never a ceiling. An edge that only just clears an adjusted bar should be trusted less than the arithmetic suggests, and an edge that clears it with room to spare has earned only a provisional pass. The correction disciplines the backtest; it does not redeem it. What settles the question is the thing the counting cannot contaminate: performance on data that did not exist when the rules were locked.
Sources & further reading
- Harvey, C. R., Liu, Y. & Zhu, H. (2016). “… and the Cross-Section of Expected Returns.” Review of Financial Studies, 29(1), 5 to 68.
- Benjamini, Y. & Hochberg, Y. (1995). “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society, Series B, 57(1), 289 to 300.
- White, H. (2000). “A Reality Check for Data Snooping.” Econometrica, 68(5), 1097 to 1126.
- Hansen, P. R. (2005). “A Test for Superior Predictive Ability.” Journal of Business & Economic Statistics, 23(4), 365 to 380.
- Politis, D. N. & Romano, J. P. (1994). “The Stationary Bootstrap.” Journal of the American Statistical Association, 89(428), 1303 to 1313.
- Bailey, D. H. & López de Prado, M. (2014). “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality.” Journal of Portfolio Management, 40(5), 94 to 107.