Try twenty indicators against last year's data. One will look great. Not because it works — because you tried twenty things and the market is noisy enough that one of them had to.
This is data mining bias, and it's the quiet reason so many strategies work right up until the moment real money is behind them.
The arithmetic
Suppose a worthless idea has a 5% chance of looking impressive on any given dataset by pure luck. Test one idea: 5% chance you fool yourself. Test twenty independent ideas: the chance that at least one looks impressive is about 64%. Test a hundred: you're essentially guaranteed a winner, and it's guaranteed to be garbage.
Nothing about this requires bad intentions. You aren't cheating. You're doing exactly what a diligent researcher does — trying things — and the arithmetic is quietly manufacturing a false positive on your behalf.
The strategy didn't find an edge in the data. You found a coincidence, and then named it.
It compounds with parameters
It gets worse. Every idea has knobs. A moving-average crossover isn't one idea — with two periods to choose, it's hundreds. Sweep them all, pick the best, and you've run hundreds of experiments and reported the winner as if it were the only thing you tried.
That's overfitting and data mining bias working together: the first fits the parameters to the noise, the second makes sure you only ever see the fit that flattered you.
The tells
The result is fragile. A 20-period lookback is great, 19 and 21 are terrible. Real effects are broad and boring; noise is sharp and specific.
You can't explain the mechanism. If you can't say why the market would pay you for this — who's on the other side, what forces them to act — you probably found a pattern, not a cause.
It only works in one slice. Only on Tuesdays. Only in 2023. Only in that regime. Real edges degrade at the edges; fake ones vanish.
The sample shrank while you weren't looking. Every filter you added carved the data smaller (sample size). The final rule may rest on nine trades.
The defences
Hypothesis first. Write down what you expect and why, then test. An idea that arrives after the data is a description of the past, not a prediction of the future.
Hold data back. Test on one period, validate on another you have never looked at. Once you've peeked, that data is spent — you can't un-see it.
Count your attempts. Honestly. If you tried thirty variations, the winner needs to clear a far higher bar than if you tried one. Most people simply don't count.
Walk it forward. Walk-forward testing at least forces the strategy to keep proving itself on data it hasn't been fitted to.
Prefer fewer rules. A simple rule with a reason beats an intricate one with a backtest. Complexity is where noise hides.
The mindset
The goal isn't to find something that worked. Something always worked. The goal is to find something that worked for a reason that will still be true tomorrow. Those are very different searches, and only one of them pays.
See also: backtesting basics and what is a trading edge.