Problem Statement

Backtesting and Multiple Testing Suppose you backtest 1000 randomly generated trading strategies on the same historical dataset and the best one shows an in-sample Sharpe ratio of 4. What do you conclude? You have discovered genuine alpha. The data set must be corrupted. Nothing yet. With 1000 trials on the same data, an extreme best-of result can arise from selection bias and multiple testing alone.