Five ways to fool yourself

1. Only the survivors

If you test a rule on today's largest companies, you are testing it on winners. Companies that fell and disappeared are missing, and they are exactly what a rule would have had to get through. This is how the 15 largest US companies of today that existed 20 years ago did:

The middle company grew 26 times, and 14 of the 15 beat the index. The whole S&P 500 with dividends grew 7.8 times. The index also held the companies that did badly and dropped out along the way. Nobody in 2006 could know which ones would end up largest.

The US companies in this course are today's largest too, and so are those in the study on technical analysis. That is why the rule is also tested on a whole index, where nothing has been sorted out.

2. A peek into the future

It is easy to use, by accident, a price that didn't exist when the decision was made. If the test owns the stock on the very day the price crosses the average, and counts that day's rise, it has looked one day ahead. On the S&P 500 since 1928, the 200-day rule returned 6.9% a year. With one day's peek ahead it returned 16.5%, and its largest fall shrank from −54% to −26%. The mistake fits in one spreadsheet cell.

3. Too many tries

Why 200 days? We tried 12 lengths of the average, from 25 to 300 days, and split the history into two halves:

The best in the first half was 150 days, at 6.2% a year. In the second half the same length came in at number 5 of 12. The best there was 200 days, which had been number 5 in the first half.

Try many variants and keep the best, and you have largely picked the one that was lucky during that time. The more tries, the better the best one looks, without the rule getting any better. That is why you set aside part of the history and don't look at it until the rule is fixed.

4. No costs

The rule made 589 buys and sells in 98 years, about 6 a year. Each trade costs a commission and the spread, and a gain is taxed. How much that takes is in the lesson on costs and tax in the test.

5. The period that suits

The same rule, decade by decade:

The rule beat holding the index in five of the ten decades: 1930s, 1940s, 1960s, 1970s and 2000s. It fell behind in 1950s, 1980s, 1990s, 2010s and 2020s. A test that covers only some of them can show almost any answer, depending on where it starts.

Looking for what supports what you already believe is confirmation bias, and feeling afterwards that the outcome was obvious is hindsight bias. Both make it easy to choose the period and the variant that look best.