p-value

A p-value is the probability of observing a result at least as extreme as the one measured, assuming the null hypothesis is true. In strategy testing the null is usually 'the rule has no edge', and the p-value comes from a permutation or bootstrap procedure on the recorded trades.

Senzoukria · Glossary · Updated September 2026

What the number means

Fix a null hypothesis, for example that the strategy's trades are indistinguishable from random draws with the same distribution. Compute a statistic on the real trades: net result, Sharpe per trade, expectancy. Then determine how often that statistic would be at least as large if the null were true. That frequency is the p-value.

In a permutation test the frequency is counted directly: rearrange the data many times, recompute the statistic, and count the rearrangements that match or beat the real one. No formula for the distribution is needed, which matters for trade results that are rarely normal.

What it does not mean

  • It is not the probability that the null is true. That would require a prior on the null, which the test does not use.
  • It is not the probability that the strategy will keep working. It describes the sample, not the future.
  • It is not a measure of effect size. A tiny edge on a huge sample can produce a small p-value; a large edge on ten trades cannot.
  • It is not corrected for selection. Pick the best of many configurations and its p-value is optimistic; the deflated Sharpe and PBO address that.

In Senzoukria

The desktop backtest panel displays a p-value in its go/no-go gauntlet, computed by the permutation test phase. The associated criterion is worded 'Permutation test: edge distinguishable from chance'. The p-value sits beside PSR, DSR, PBO and the number of configurations tried, so a reader sees at once whether the sample also survives the selection correction.

The Replay performance panel does not report a p-value; it flags small samples with the note that the ratios are shown because they are exact, not because they are significant. Significance testing is a backtest-panel function.

Common mistakes

  • Stopping at a fixed threshold. A conventional cut-off is a convention, and a false positive is paid for on a live account rather than in a paper.
  • Testing repeatedly on the same data and keeping the smallest p-value. Each test is a trial.
  • Reporting a p-value without the statistic and the sample size that produced it.
  • Reading a large p-value as proof of no edge. It only says this sample did not show one.

This page in other languages

Frequently asked questions

Is a small p-value enough to trade a strategy?
No. It says the observed result was rare under the no-edge hypothesis on that sample with those assumptions. It does not account for costs that were not modeled, for the number of configurations tried, for drawdown against account rules or for regime change. The desktop gauntlet lists it as one criterion among several for that reason.
Why does the p-value change when I rerun the gauntlet?
The permutation phase draws random rearrangements, so two runs with the same data produce slightly different counts. The variation shrinks as the number of permutations grows. If the p-value jumps across a decision boundary between runs, the sample is telling you the result is marginal.

Keep reading

Strategy researchTest a trading strategy, from observation to challenge rulesDefine an observation, question it with the copilot, write a rule and explore costs, drawdown and fictional challenge limits in an interactive example.Quant · Start hereQuant trading for beginners: from idea to backtestDefine a testable rule, choose suitable data and evaluate later periods after costs.Order flow · HubOrder flow trading: the complete guideThe hub. What order flow is, what it shows — aggression, delta, imbalance, absorption, liquidity — and the tools that read it, each linked to a full guide.