Probability of backtest overfitting (PBO)
The probability of backtest overfitting (PBO) estimates how often the configuration that performs best in sample performs below the median of all configurations out of sample. It is computed by splitting the full-period results of a parameter grid into many train/test combinations.
Senzoukria · Glossary · Updated September 2026
How PBO is estimated
Start with a matrix: one row per time slice, one column per configuration, each cell holding the result of that configuration on that slice. Partition the slices into an even number of groups. For every way of choosing half the groups as training, rank the configurations on the training half, keep the best one, and look up its rank on the remaining half.
If the in-sample winner ranks below the median out of sample, that combination counts as an overfit. PBO is the fraction of combinations where this happens. The method, called combinatorially symmetric cross-validation, was published by Bailey, Borwein, López de Prado and Zhu.
- A PBO close to zero means the in-sample winner usually stays above the median out of sample.
- A PBO close to one half means selecting the best configuration is no better than picking one at random.
- Above one half the in-sample winner tends to be an out-of-sample loser, which points to a selection process that rewards noise.
What PBO does not say
PBO is a statement about the selection procedure on this grid and this period. It does not measure the size of the edge, and a low PBO on a grid of losing configurations only means that the least bad one stays the least bad. It also inherits every assumption of the underlying backtest: fill policy, costs, coverage gaps and bar construction.
In Senzoukria
The backtest panel of the desktop runs a full-period grid after the walk-forward: the note reads 'Full-period grid: N configurations rerun on all bars for the PBO matrix.' The gauntlet then lists 'Probability of backtest overfitting' as a criterion and shows the PBO value beside the PSR, DSR and p-value. The panel introduces the block as 'the probability that picking the best in-sample config hurts out of sample'.
Because PBO needs the whole grid on the whole period, it is only available after a sweep. A single configuration produces no matrix and no PBO.
Common mistakes
- Comparing PBO values across grids of different sizes as if they were on the same scale.
- Using PBO as a stand-alone verdict. It says nothing about drawdown, costs or account rules.
- Reducing the grid to a few configurations to obtain a low PBO. The estimate becomes coarse and the selection bias is hidden elsewhere.
- Overlooking that the slices are contiguous in time: a regime confined to one slice can dominate the ranking.
Related
This page in other languages
Frequently asked questions
- What PBO value counts as acceptable?
- There is no universal cut-off. The value is a probability that the in-sample winner underperforms the median out of sample, so anything near one half means selection adds nothing. Read it together with the walk-forward efficiency and the deflated Sharpe, and keep the grid size in mind because a coarse grid produces a coarse estimate.
- How is PBO different from walk-forward testing?
- Walk-forward advances chronologically and reports the result of frozen parameters on later windows. PBO reuses the entire grid on every combination of slices, in both directions, to measure how often the selection rule picks a configuration that fails out of sample. Walk-forward measures performance; PBO measures the reliability of the selection.