Permutation test

A permutation test checks whether a backtest result could have arisen by chance by recomputing the same statistic many times on data whose structure has been randomly rearranged. The share of permutations that match or beat the real result is the p-value.

Senzoukria · Glossary · Updated September 2026

Building the null distribution

The question is not 'did the strategy make money' but 'would a rule with no information have made this much'. To answer it, destroy the information the rule could exploit while keeping everything else: the same bars, the same number of trades, the same costs. Rearranging the order of the returns, or the sign of each trade, or the pairing between signals and outcomes are the usual ways to do that.

Recompute the objective on each rearranged series. After many permutations you have a distribution of results that a rule without edge would produce on this exact data. Place the real result on that distribution.

Reading the outcome

  • The p-value is the fraction of permutations whose statistic is at least as good as the observed one.
  • A small p-value means few random rearrangements reached the observed result; the edge is distinguishable from chance on this sample.
  • A large p-value means random rearrangements reach it routinely. The result may still be real, but this data does not show it.
  • The test is exact for the sample and needs no distributional assumption, which is why it suits fat-tailed trade results.

In Senzoukria

The go/no-go gauntlet of the desktop backtest panel lists 'Permutation test: edge distinguishable from chance' as one of its criteria and shows the resulting p-value alongside PSR, DSR and PBO. The panel introduces the block as 'a permutation test against pure chance', run in a phase labelled gauntlet · permutations after the walk-forward and the full-period grid.

The site's backtest page lists permutation tests with PSR, DSR and PBO among the research tools available on supported historical tests. Like the other criteria, the test uses the same fill policy and costs as the backtest that produced the trades.

Common mistakes

  • Permuting in a way that also destroys the structure the strategy legitimately uses, such as session boundaries, which makes the null too easy to beat.
  • Running the test on the in-sample winner of a sweep without accounting for the trials. The permutation p-value does not correct for selection; the deflated Sharpe does.
  • Reading the p-value as the probability that the strategy has no edge. It is the probability of the observed result under the null, not the other way around.
  • Using too few permutations, which makes the smallest reportable p-value coarse.

This page in other languages

Frequently asked questions

How many permutations are needed?
Enough that the p-value resolution is finer than the decision you want to make. With one hundred permutations the smallest non-zero p-value is one in a hundred. The desktop runs the permutations as a separate gauntlet phase after the walk-forward; the count is set by the panel and displayed with the result.
Can a permutation test replace walk-forward validation?
No. The permutation test asks whether the result on the tested data is distinguishable from chance. Walk-forward asks whether frozen parameters keep working on later data. A strategy can pass one and fail the other. The gauntlet reports both because they fail for different reasons.

Keep reading

Strategy researchTest a trading strategy, from observation to challenge rulesDefine an observation, question it with the copilot, write a rule and explore costs, drawdown and fictional challenge limits in an interactive example.Quant · Start hereQuant trading for beginners: from idea to backtestDefine a testable rule, choose suitable data and evaluate later periods after costs.Order flow · HubOrder flow trading: the complete guideThe hub. What order flow is, what it shows — aggression, delta, imbalance, absorption, liquidity — and the tools that read it, each linked to a full guide.