This article explains how backtests mislead and how to read past it. It is educational, about evaluating any track record, not personalised investment advice, and not a performance claim about any specific strategy.
A backtest is the easiest thing in finance to fake and the hardest thing to fake honestly. Almost any rule, tuned enough, can be made to produce a gorgeous historical equity curve, and almost none of those curves survive contact with live markets. The gap between the two is not bad luck. It is a small number of specific, mechanical errors, each of which inflates the past at the expense of the future. Knowing the errors is how you read a backtest for what it is rather than what it advertises.
Look-ahead bias: using information you wouldn’t have had
The most common and most lethal error is look-ahead: the backtest makes a decision using information that was not available at the moment of the decision. It is usually subtle. Using a day’s closing price to decide a trade you mark as entered at that day’s open. Using a company’s restated earnings rather than the numbers actually published at the time. Ranking a universe using a list of names that was only finalised later. Each one quietly hands the strategy a peek at the future, and even a tiny peek compounds into a fantasy curve.
Look-ahead is dangerous precisely because it doesn’t feel like cheating. The author isn’t trying to lie; the data pipeline leaks the future and the result looks brilliant. The only defense is a discipline of point-in-time data, every decision uses only what was knowable on the day, including which names even existed in the universe then. That last point is its own deep problem, the survivor leak, covered in survivorship bias.
Overfitting: fitting the noise instead of the signal
The second error is overfitting, tuning a strategy so precisely to the historical data that it captures the random noise of that specific period rather than any durable edge. Every added parameter, every threshold nudged to improve the curve, every “the strategy works except in these three months so let’s add a filter” is a step further into describing the past instead of predicting the future.
The tell is fragility. An overfit strategy’s performance collapses if you shift the test window by a month, change a threshold by a hair, or run it on data it wasn’t tuned on. A real edge is robust: it degrades gracefully when you perturb it, because it was capturing something structural, not a coincidence. A curve that is too perfect, no ugly stretches, every parameter suspiciously round, is more worrying than a good-but-scarred one.
The in-sample / out-of-sample split
The first defense against overfitting is to never judge a strategy on the data you built it with. Split history: develop and tune on one portion (in-sample), then test, once , on a portion the strategy has never seen (out-of-sample). If the edge survives on data it wasn’t fitted to, it has a chance of being real. If it only shines in-sample, it was memorisation.
The catch is that “once” is doing heavy lifting. The moment you peek at the out-of-sample result, dislike it, and go back to tweak, the out-of-sample data has become in-sample, you are now fitting to it too, just more slowly. Honest out-of-sample testing requires the discipline to accept a disappointing result rather than quietly tuning until it passes. That self-discipline failure is also why the bar for statistical significance has to be set far higher than the textbook one: when many variants are tried, some pass by luck alone.
Walk-forward: the closest a backtest gets to honest
Walk-forward validation is the more rigorous version of the split. Instead of one in-sample/out-of-sample division, you roll it: tune on a window, test on the next unseen window, then slide both forward and repeat across the whole history. The strategy is always being judged on data just after the data it was tuned on , which is exactly the situation it faces live, where you always trade the period just after the one you learned from.
Walk-forward doesn’t make a backtest truthful, nothing fully does, but it removes the easiest ways to fool yourself. A strategy that holds up walk-forward, across many rolling windows including ugly ones, is making a far stronger claim than one with a single beautiful in-sample curve. And the ultimate walk-forward is live: running the locked rules forward in public, where the next window is real and unseen by construction. That is why we paper-trade in public and why the day the live bot caught up to its backtest mattered: live is the test that can’t be overfit.
Reading a backtest you didn’t build
You usually can’t audit someone else’s pipeline, but a few questions surface most of the lies:
- Is the curve suspiciously smooth? Real edges have ugly stretches. A flawless curve is a red flag, not a selling point.
- How many parameters, and how round are they?A dozen finely-tuned thresholds is a description of the past. A few robust ones that work across a range is an edge.
- Is there out-of-sample or walk-forward evidence?Or is the whole claim one optimised pass over all of history?
- Is any of it verified forward? A backtest is a claim about the past. Live, public, unedited results are the claim being tested where overfitting can’t reach.
Most backtests don’t lie on purpose. They lie because the person who built them wanted the strategy to work and the data was happy to oblige. The defense is not cynicism, it is the small set of disciplines above, applied to your own work before you trust it, and demanded of anyone else’s before you do.
Sources & further reading
- Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). “Pseudo-Mathematics and Financial Charlatanism.” Notices of the American Mathematical Society, 61(5), 458 to 471.
- López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.