Backtests lie.
Test enough trading rules on the same history and the best one looks brilliant by luck alone. Here is how brilliant, and what happens next.
- rules tested
- 4,916
- best in-sample Sharpe, on average
- 1.17
- out-of-sample Sharpe of that rule
- 0.03
- effective independent trials
- 124
- real EUR/USD walk-forward Sharpe
- −0.02
Frozen run, five years of simulated normal returns with zero expected return, 500 draws. An educational demonstration, not a finding about any market and not advice.
Pick the best of N rules. Then watch it fail.
Each draw simulates a price history with no edge, scores every rule on the first part, picks the best of a random subset of N rules, and then trades it on the part it has never seen. Everything runs in your browser; the same seed gives the same result.
Run the experiment to see the numbers.
Show the numbers as a table
| Rules tested | Best in-sample | Out-of-sample |
|---|
The frozen run, and what was predicted.
Before any result existed, the protocol fixed the rule family, the return models, the sample lengths and five hypotheses with decision rules. This is what 500 draws per cell showed, with five years used to choose and five to test.
| Return model | Rules tested | Best in-sample | That rule out-of-sample | Independent reference | Effective trials |
|---|---|---|---|---|---|
| Normal returns | 1 | −0.00 (−0.74 to 0.77) | −0.03 (−0.76 to 0.70) | 0.00 | 1 |
| Normal returns | 10 | 0.60 (0.21 to 1.12) | −0.02 (−0.71 to 0.69) | 0.70 | 6 |
| Normal returns | 100 | 0.88 (0.51 to 1.31) | 0.01 (−0.69 to 0.72) | 1.13 | 22 |
| Normal returns | 1,000 | 1.06 (0.71 to 1.49) | 0.00 (−0.70 to 0.73) | 1.46 | 64 |
| Normal returns | 4,916 | 1.17 (0.80 to 1.61) | 0.03 (−0.69 to 0.70) | 1.65 | 124 |
| Fat-tailed returns | 1 | 0.01 (−0.72 to 0.74) | 0.01 (−0.76 to 0.78) | 0.00 | 1 |
| Fat-tailed returns | 10 | 0.60 (0.22 to 1.05) | −0.03 (−0.74 to 0.67) | 0.70 | 6 |
| Fat-tailed returns | 100 | 0.87 (0.51 to 1.29) | 0.00 (−0.68 to 0.66) | 1.13 | 22 |
| Fat-tailed returns | 1,000 | 1.06 (0.73 to 1.50) | −0.00 (−0.73 to 0.78) | 1.46 | 64 |
| Fat-tailed returns | 4,916 | 1.15 (0.84 to 1.57) | −0.01 (−0.78 to 0.69) | 1.65 | 114 |
| Volatility clustering | 1 | 0.04 (−0.68 to 0.76) | 0.03 (−0.70 to 0.75) | 0.00 | 1 |
| Volatility clustering | 10 | 0.59 (0.24 to 1.06) | −0.03 (−0.79 to 0.74) | 0.70 | 6 |
| Volatility clustering | 100 | 0.86 (0.53 to 1.29) | −0.01 (−0.79 to 0.76) | 1.13 | 20 |
| Volatility clustering | 1,000 | 1.05 (0.69 to 1.50) | −0.01 (−0.67 to 0.71) | 1.46 | 58 |
| Volatility clustering | 4,916 | 1.15 (0.80 to 1.58) | −0.02 (−0.78 to 0.72) | 1.65 | 110 |
Five predictions, scored as written
- H1: Selection inflates the in-sample Sharpe ratioSupported. The best in-sample Sharpe at 4,916 rules exceeds the single-rule value by 1.17, 1.14, 1.10 (normal, fat-tailed, volatility clustering) against a threshold of 0.6.
- H2: The selected rule has no out-of-sample edgeNot supported as written. 1 of 54 cells lie beyond two standard errors of zero; about 2.5 are expected by chance alone with no correction.
- H3: Rules are strongly correlated, so far fewer than N are independentSupported. The effective number of independent trials is below the number of rules in every cell with 100 or more rules.
- H4: In-sample rank does not predict out-of-sample rankSupported. The mean rank correlation between in-sample and out-of-sample Sharpe is within 0.05 of zero in every cell.
- H5: The real EUR/USD walk-forward looks like the sign-flip nullSupported. The real-series walk-forward result lies inside the central 90% range of the sign-flip null.
The frozen decision rule for H2 had no multiplicity correction, so one exceedance out of 54 comparisons counts as not supported even though chance alone predicts about two or three. Both facts are reported.
The same trick on real EUR/USD.
Every year, pick the best of all the rules on the previous five years of the ECB's daily EUR/USD reference rate, then hold it for the next year. Stitch the held-out years together. As a yardstick, repeat the whole procedure on 200 copies of the series whose daily returns had their signs flipped at random, which keeps the volatility pattern and removes any trend structure.
The selected rules looked good, then did nothing.
- average in-sample Sharpe of the selected rules
- 1.10
- out-of-sample Sharpe, stitched
- −0.02
- block-bootstrap 90% range
- −0.37 to 0.32
- range of 200 sign-flip nulls
- −0.37 to 0.32
- buy-and-hold Sharpe, same dates
- −0.08
22 yearly selections from 2004-11-26, 5,544 out-of-sample days.
Limits: what this does not show.
These are as important as the result. The full list is in the repository.
- Luck is built in. The simulated markets have no edge by construction, so "the best rule is lucky" is true by design. The experiment measures how big the luck effect is. It says nothing about whether any real strategy works.
- One rule family, one real series. Moving-average crossovers and momentum, and EUR/USD only. The rules overlap strongly: roughly 125 independent trials stand in for 4,916 rules (normal returns, five years).
- No costs. Transaction costs, financing and market impact are ignored. They would make every result look worse.
- Sharpe ratios are noisy. Five years of daily data give a standard error near 0.45, and the answers depend on windows, split and seed.
- Browser and registry differ in detail. The browser uses its own random numbers: single runs differ from the frozen registry, the distributions agree.
- Not advice. This is an educational demonstration of selection bias and multiple testing. It makes no claim about the predictability of any market.
Reproduce it yourself.
The Python package needs only NumPy. The ECB series is in the repository with its checksum, and the protocol was committed before the first result.
pip install -e .
pytest
python scripts/generate_registry.pyHow it was built
Designed and directed by David Lindgreen. The code, tests and text were written with substantial assistance from an AI system (Claude, Anthropic). Every number on this page is generated from the frozen registry, and the author is responsible for the content.
Method references
Bailey and López de Prado, "The Deflated Sharpe Ratio," Journal of Portfolio Management 40(5), 94-107 (2014). Bailey, Borwein, López de Prado and Zhu, "The Probability of Backtest Overfitting," Journal of Computational Finance 20(4) (2017).