The backtesting period should be relevant in proportion to the average holding period of the trades. From a statistical standpoint, as the number of periods increases the confidence in the measurement increases (but at a diminishing rate). As a simple rule of thumb, we would not consider a strategy with less than 20 samples. So for an average holding period of one year, we would not consider strategies with less than a couple decades of samples. If you have multiple strategies with low trade counts but that share a common factor but are otherwise sufficiently independent, that could allow you to consider them together.

Having recent backtesting is also very important, as the market changes over time and recent measurements have more predictive value than long ago measurements. We have research on autocorrelation of returns that demonstrates this empirically. Using recent data has to be balanced against the competing goal of keeping some recent data hidden from the model (out of sample) so you can reduce overfitting (a.k.a. curve fitting) and measure the true goal of how the model performs into the future (or out of sample period), rather than making a perfect model of the past and a poor model of the future.

Having a period with a variety of market conditions is also valuable if you are trying to design a strategy that works in all market/economic cycles. Economic research suggests a variety of business cycle lengths, including between 7-11 years, so depending on holding periods, you might want to cover at least one cycle and preferably multiple.

Finally, we would recommend backtesting against a variety of periods. If a strategy only produces good results for one specific time period and poor results otherwise, that is evidence that the strategy is too fragile and cannot handle a variety of possible future market conditions. So we by backtesting a variety of time periods, and comparing the results, and perhaps pick a model that has lower but more consistent performance across a variety of time periods, rather than a strategy with great performance but only for one period and poor performance in other periods.

There are multiple ways you could define "best" performance. Our preferred measure compares risk vs return is called the Sharpe Ratio, which was devised by Nobel Prize Winner William Sharpe. We prefer that measure over just return for example as we have found that the strategies with the highest return tend to be the strategies with the highest risk. So by choosing highest return you may be inadvertantly choosing strategies with highest risk. For our scanner, we modify the standard Sharpe Ratio by discounting it for strategies with a low number of trades (or samples). We build a confidence interval around the Sharpe Ratio and use a 95% confidence level of the lowest estimate.

The Sharpe Ratio we calculate is not the exact value but an estimate of the true value based on a finite number of samples. Essentially our confidence around our estimate increases as the number of samples increases. To compute the 95% confidence interval around the Sharpe Ratio, using the equation here. See equation (10) in this paper for more detail. As an example, if your average return per trade is 1.0% and your standard deviation is 1.0%, your estimated Sharpe Ratio is 1.0 based on 10 trades. The 95% confidence level of the worst case true value of the Sharpe Ratio would be: 1.0 - 1.96 * SQRT((1 + 0.5 * 1.0^2) / 10) = 0.24. In other words you have a 97.5% confidence that the true Sharpe Ratio is at least 0.24 based on a 10 sample estimate. (97.5% greater than x is equivalent to a 95% confidence between x and y).

A good book on our shelf relevant to financial statistics is "Analysis of Financial Time Series" by Ruey Tsay . He is a PhD in Statistics who teaches at U of Chicago Booth business school. It is an academic textbook and probably designed to be read by mathematicians. The world of finance does not fit in a perfect mathematical box but it still provides some of the most powerful tools. We also like Quantpedia which writes concise summaries of academic papers in finance for an easier, faster read.