How to run your own test

Seven steps

1. Write the rule so someone else could follow it. What you buy, when you buy, when you sell, at which price and how much. Don't change it after seeing the result, or count the change as a new try.
2. Decide what to compare it with. Usually buying the same stock or index on the same day and holding it, over the same time.
3. Use data where nothing has been sorted out. A whole index, or the companies that existed then, not only today's winners. Why is in the lesson on five ways to fool yourself.
4. Trade the day after the signal. A signal computed on the closing price only shows once the day is over.
5. Count costs and tax. Higher than you expect, as in the lesson on costs and tax.
6. Keep part of the history aside. Try variants on the first part, and run the rule you chose on the last part only once.
7. Compare with chance. See below.

Compare with chance

A rule can beat holding by pure luck. One way to see it is to let rules that buy and sell at random do the same. Here are 1,000 random rules on the S&P 500. Each buys 295 times, like the 200-day rule, and owns the index for the same share of the time, 68%:

0 of the random rules made more than the 200-day rule. So the rule did something chance rarely does: it was out more often when the index fell a lot, and its largest fall was −54% against −86% for holding.

Only 16 of the random rules beat holding the index. Being out part of the time usually misses part of the rise, however the days are chosen.

Beating chance is not the same as beating buy and hold after costs. Before costs, the 200-day rule returned 6.9% a year and holding 6.1%. At 0.25% per trade it was 5.3% against 6.1%.

What a test can't show

A test shows how the rule would have done, with the prices and costs you counted. It doesn't show how it does from here, and a result from a handful of trades can come down to a few single days. The more trades and the more periods the rule holds up in, the less the result depends on luck. How five well-known signals did in the same kind of comparison with an ordinary day is in the study on technical analysis.