Backtesting Mistakes: Bias, Costs and Overfitting, With Examples

Backtests that look good are often flattered by one of a handful of mistakes: costs left out, information from the future, rules tuned to the sample, or too few trades to mean anything. Each one makes the past look easier than it was. This page lists them, with a worked example where a number helps. All inputs are made up, and every figure is computed from them.

1. Leaving out the costs

The cost of a trade is not a fixed number of R: it is the spread (plus any commission) divided by how far away your stop is. The same 1.2 pips of spread is a big share of a small stop and a small share of a wide one. Take a made-up rule that wins 45% of the time for 1.8R and loses the rest for 1.0R. Before costs its expectancy is 45% × 1.8 − 55% × 1.0 = +0.26R per trade. After the spread, by stop distance (a simplification: it treats the spread as a straight cost taken from each trade):

The same rule, with the same spread, goes from +0.02R a trade at a 5-pip stop to +0.23R at a 40-pip one. Short stops are not worse by nature; they simply need a better result before costs to end up in the same place. A backtest that skips the cost reports the +0.26R for every row.

2. Using information you would not have had

Look-ahead is acting on something that was not yet known. The commonest form: a rule that needs a bar's close gets filled at that bar's open. Made-up case: a bar opens at 1.1000 and closes at 1.1030, and the rule says “enter when the bar closes above the recent high”. You can only know that at the close. Entering at the open instead hands the trade 30 pips it could not have had, which on a 10-pip stop is 3.0R. Repeated across a test, that is an edge that was never there.

• Enter on the bar after the one that confirmed the entry rule, at its open, not on the confirming bar.
• Check every indicator value you use was computable at that moment. Some redraw as new bars arrive.
• In a hand backtest, cover the chart to the right (see the steps).

3. Tuning the rules to the sample

Try ten values for each of three settings and you have tried 1,000 combinations. Report the best one and you have reported the best of 1,000 draws against the same history. By chance alone, the best of many looks better than a typical one, and it says little about the next stretch of data. Keep the number of settings small, say how many you tried, and test the chosen one on data that played no part in choosing it (see backtest, forward test and demo).

4. Too few trades

With the same made-up rule, one trade's result has a standard deviation of 1.39R, so an average over 20 trades has a standard error of 1.39 ÷ √20 = 0.31R. A rough band of two standard errors either side is ±0.62R, wider than the +0.26R the rule truly averages. Twenty trades could show it losing or winning well. The longer treatment is in how many trades before your stats mean anything.

5. Choosing the period afterwards

Pick the stretch of history after seeing how the rules did on it and you have chosen the answer. Decide the market, the timeframe and the dates first, write them down, and include conditions that suit the rules and conditions that do not.

6. Assuming every touched price was a fill

A bar that reaches your limit price does not tell you that your order would have been filled; price can touch a level without an order placed there executing. A test that fills every touch is more generous than trading. Mark such fills as an assumption, or leave limit entries out of a first test.

7. Not knowing what happened inside the bar

When one bar's range reaches both a stop and a target, the bar does not say which came first. A fixed rule for those bars, written down, is better than a different guess each time. The worked example is in bar replay against tick data.

8. Changing the rules after looking at the results

Every adjustment made after seeing the outcome lets the sample shape the rules. If you change one, count it as a new test on new data, and keep the earlier result alongside it so you can see how much each change moved it.

9. Believing the tester's mode

Automated testers trade accuracy for speed by mode, and the quick modes build ticks out of bars. A result from a quick mode is a first look. See manual against automated backtesting for what the documented modes do.

A checklist to run before you trust a result

• Were the rules written before the test, and unchanged during it?
• Were the market, timeframe and dates chosen first?
• Is every cost in, expressed in R for each trade's own stop?
• Could every input have been known at the moment of entry?
• How many settings were tried, and was the winner checked on data it never saw?
• How many trades are there, and what is the standard error of the average?
• How were ambiguous bars and limit fills treated, and is it written down?

None of this makes a rule profitable, and a test that passes the checklist can still fail on the next trade. The figures on this page are arithmetic on made-up inputs, not advice.

Trepeat today

Trepeat does not sell a backtester.

We're working on a free journal and a backtester, with no date yet. Today Trepeat is a trade copier for MT4, MT5 and cTrader, currently on demo accounts, and you can add notes, a grade and screenshots to any copied trade once it closes, with a calendar of your results.

How the journal works. The lot size calculator gives the position size for a given stop and amount at risk, and the backtest log template has a column for each trade's cost.

Written by the Trepeat team.