Backtesting: what it proves, and what it hides

A backtest answers one question: if these exact rules had run over this exact period, what would have happened? That is genuinely useful. The trouble is that people hear it answering a much broader question — will this make money — and it cannot answer that at all.

Nearly every backtest is too optimistic, and usually for reasons that are specific, well understood, and correctable. Here are the ones that do the most damage, roughly in order of how much they inflate results.

Lookahead bias

Lookahead bias means using information that would not have been available at the moment of the decision. It is the most destructive error because it can be invisible in code that looks perfectly reasonable.

# Wrong — decides using the close of the candle it trades on
df["signal"] = df["close"] > df["sma_20"]
df["ret"] = df["signal"] * df["close"].pct_change()

# Right — the decision uses only completed information
df["signal"] = (df["close"] > df["sma_20"]).shift(1)
df["ret"] = df["signal"] * df["close"].pct_change()

The first version buys at a candle's close using knowledge of that same close. In live trading you cannot know a candle's closing price until it has closed, at which point you can only act on the next one. That single missing shift can turn a mediocre strategy into a spectacular one on paper. If a backtest produces an unusually smooth equity curve, this is the first thing to check.

Survivorship bias

If you test a strategy on the current NIFTY 50 constituents over the last ten years, you are testing it on companies that were successful enough to still be in the index today. The ones that were removed after collapsing are absent from the data, and your strategy never had to hold them on the way down. Index membership changes over time, and a test that ignores that is measuring the index committee's hindsight as much as your rules.

Costs and slippage

This is the one that most reliably kills intraday strategies. Every trade in India carries brokerage, STT, exchange transaction charges, GST, SEBI turnover fees, and stamp duty. On top of those explicit costs sits slippage — the difference between the price your signal saw and the price you actually got.

A strategy averaging 0.15% per trade before costs, trading ten times a day, is not a good strategy. It is a mechanism for transferring money to intermediaries. The arithmetic is unforgiving and it is the single fastest sanity check available: take your average gross edge per trade, subtract a realistic round-trip cost, and multiply by your trade frequency. If that number is not comfortably positive, no amount of further optimisation will save it.

• Model costs per trade explicitly, not as an annual percentage.
• Assume you get filled at the next candle's open, not the signal candle's close.
• For anything illiquid or large relative to typical volume, add slippage well beyond the spread.
• Test whether the strategy survives costs twice as high as your estimate. If it does not, the margin is too thin to be real.

Overfitting

Given enough parameters and enough attempts, any dataset can be fitted perfectly. A strategy with six tunable numbers, optimised over three years of data, will find a combination that looks superb — and that combination describes the noise in those particular three years rather than any durable market behaviour.

The defences are unglamorous and effective: prefer fewer parameters; be suspicious of results that depend on precise values; and check the neighbourhood. If a 20-period lookback is profitable but 18 and 22 both lose money, you have not found an edge, you have found a coincidence. A genuine edge tends to degrade gracefully as parameters shift rather than falling off a cliff.

Walk-forward testing

The most useful structural improvement is to stop optimising and testing on the same data. In walk-forward testing you fit parameters on one window, test on the next unseen window, then roll both forward and repeat. What you end up with is a record of how the strategy performed on data it had never been tuned against — which is the only situation it will ever face live.

Results from walk-forward testing are almost always worse than from a single optimised backtest. That is the point. The number is lower because it is closer to true.

What a trustworthy backtest looks like

By the time a strategy is worth risking capital on, the test behind it should include realistic costs and slippage, use only information available at decision time, run across regimes that include at least one serious drawdown, report the worst peak-to-trough decline alongside the return, and show results on data never used for tuning. It should also be boring. A backtest that looks too good is not usually a discovery — it is usually a bug, and finding it is cheaper before the capital goes in than after.