Overfitting tests: deflated Sharpe, PBO, permutation and stability

Senzoukria measures whether a backtest result survives the number of configurations tried: probabilistic and deflated Sharpe ratios, the probability of backtest overfitting (CSCV), a sign-flip permutation test, an order-shuffle drawdown distribution and a parameter stability check. All are computed locally after an automatic backtest.

Senzoukria · Documentation · Updated September 2026

Where to find it

Where
Replay → Automatic backtest → result → Gauntlet block and the "Deflated Sharpe — probability of a real edge" line of the walk-forward
References
Bailey & López de Prado (2014) for PSR/DSR; Bailey, Borwein, López de Prado & Zhu (2015) for PBO
Permutations
10,000 draws, seed 0, deterministic
PBO
16 contiguous blocks by default, ≥ 8 session days, ≥ 2 configurations

What each test answers

Statistics shown in the Gauntlet block, their input series and the reading thresholds
StatisticInput and methodThreshold
PSRDaily P&L of the chosen path; probability that the true Sharpe exceeds 0, corrected for sample length, skew and kurtosis; 0 below 5 observationsInformational
DSRSame series, but the benchmark is the Sharpe expected from the best of N configurations without an edge, scaled by the variance of the grid's Sharpes> 0.95 to pass (eliminatory)
p-valuePer-trade P&L; sign of each trade flipped at random 10,000 times; p = (1 + count ≥ observed) / (1 + n)< 0.05 to pass (eliminatory)
PBOSession-day × configuration matrix; for every half-split of 16 blocks, rank out of sample of the best in-sample configuration< 0.5 to pass (advisory)
configurations triedNumber of grid combinations (N), the entire sweep, not only the kept onesFeeds DSR
Drawdown, 95 % of orderings do betterPer-trade P&L reshuffled 10,000 times; 5th percentile of max drawdownInformational
Parameter stabilityGrid Sharpe of the chosen configuration vs neighbours one step away on one axisplateau ≥ 0.7 median ratio and no negative neighbour; peak < 0.4 or a negative neighbour

The walk-forward line

Independently of the Gauntlet, the walk-forward result carries its own deflated Sharpe (lib/quant/overfit.ts) computed on the out-of-sample trades. It shows the Sharpe per trade against the threshold produced by selection alone, over the configurations tried and the out-of-sample trade count, and reads the probability in one word: solid at ≥ 0.95, fragile between 0.5 and 0.95, noise below 0.5.

The two implementations differ on purpose. The walk-forward line works on per-trade returns and uses the standard deviation of all Sharpes tried across folds as the spread. The Gauntlet works on session-day P&L and takes the spread from the full-period grid. A run without a sweep has N = 1 and zero variance, so the DSR reduces to the PSR against zero, which is what it must be when nothing was selected.

How to use it

  • Read the p-value first: if the sign-flip test cannot distinguish the Sharpe from chance, the rest is moot.
  • Then the DSR: a high raw Sharpe with a DSR under 0.95 means the number of configurations explains a good part of the result. Narrow the sweep or lengthen the period.
  • PBO near 0.5 is chance; above it, choosing the best in-sample configuration actively hurts out of sample.
  • Prefer a plateau over a peak in the stability chart: a rule that works around its parameters survives the market moving one step on its own.

Limits or pitfalls

These tests correct for selection and non-normality inside the sample you gave them. They do not see data that was never loaded, a strategy edited after a bad verdict, or a second sweep run on the same bars. Each rerun is another draw the counter does not know about.

A degenerate P&L (fewer than two trades or zero standard deviation) returns a p-value of 1 and a Sharpe of 0. Fewer than five daily observations return PSR = 0. These are refusals to conclude, not measurements of a bad strategy.

Passing every test is a statement about a sample; it is not a promise of live results.

This page in other languages

Frequently asked questions

Why is PBO shown as "—"?
The grid had a single configuration, or fewer than 8 session days were available for the matrix. PBO needs at least two configurations and enough contiguous blocks to split in half.
Why is the DSR lower than the PSR?
The PSR benchmarks against zero; the DSR benchmarks against the Sharpe the best of N random configurations would reach. With more trials or a wider spread of grid Sharpes, that benchmark rises and the DSR falls.
Are the permutation results reproducible?
Yes. The generator is the same mulberry32 used by the prop firm engine, seeded at 0 by default, so the same trades give the same p-value and percentiles on every run.

Keep reading

Strategy researchTest a trading strategy, from observation to challenge rulesDefine an observation, question it with the copilot, write a rule and explore costs, drawdown and fictional challenge limits in an interactive example.Quant · Start hereQuant trading for beginners: from idea to backtestDefine a testable rule, choose suitable data and evaluate later periods after costs.Order flow · HubOrder flow trading: the complete guideThe hub. What order flow is, what it shows — aggression, delta, imbalance, absorption, liquidity — and the tools that read it, each linked to a full guide.