FAQ About Backtesting Parameters

Is it better to backtest ETFs than individual stocks?

Regarding ETFs vs individual stocks, one of the biggest differences is around volatility. ETFs are much more diversified and so in general have lower volatility than individual stocks. If your goal is large returns, volatility is often an important ingredient and individual stock strategies may outperform ETFs. If your goal is less risk, then the diversification and lower volatility of ETFs will probably yield better risk adjusted returns (e.g. Sharpe Ratio.)

Should I Backtest Multiple Securities Per Strategy?

Having more securities in a test can increase the return on initial capital for strategies that sit in cash much of the time, as they can produce more opportunities for the capital to work more often. For strategies who's capital is already working most of the time, adding more securities to the strategy may or may not increase returns or decrease risk. Having securities that are correlated and have similar volatility is logical as they are more likely to react to a strategy in a similar manner.

What timeframe length should I test intraday strategies that trade many times per day?

A statistician would say it is not about time, it is about sample size. The more samples the more confidence. A strategy with one trade over twenty years would have less statistical confidence than an intraday strategy with 100 trades in 6 months. A simple rule of thumb would be to require at least 10-20 positions (a.k.a. round trip trades) to even consider a strategy. To quantify, you can calculate a confidence interval around return calculations similar to how pollsters in elections do when they report a candidate is estimated to have 40% of the vote +/- 4%. Here is more detail about confidence intervals. Beyond sample size, 10 years is a good minimum time length for once per day type strategies as you are likely to capture both bull and bear markets as economic cycles are often 7-11 years in length. The SEC also requires a minimum of 10 years for backtests that are marketed to institutional investors for investment. Strategies that trade many times per day it is often impossible to go back that far as databases typically get overwhelmed with the exponential growth in data storage. For example, 10 years of daily data is the same amount of data as 6 days of one minute data. For sample size reasons, going back 10 years for an intraday strategy is not only impractical but perhaps unnecessary. You can think of intraday strategies having a much shorter lifespan perhaps like a fruit fly rather than a human. Measuring daily strategies over the same length of time as an intraday strategy with many more trades is applying a higher hurdle on a trade sample size basis.

How long of a time frame should I test daily strategies that trade once per day?

Answer here