In-sample vs out-of-sample
In-sample data is the historical period used to choose a strategy's rules and parameters; out-of-sample data is a later period that played no part in that choice and is used only to evaluate the frozen result. The distinction matters because performance measured on the data that selected the parameters overstates what the same parameters would have delivered on data they had not seen.
Senzoukria · Glossary · Updated September 2026
Two jobs for two periods
The split has to be decided before parameters are examined. If the boundary is moved after seeing results, or the test period is used to pick between two finalists, it has become in-sample data regardless of its label.
| Period | Used for | Must not be used for |
|---|---|---|
| In-sample (training) | Choosing rules, thresholds, parameters | Reporting performance as if it were achievable |
| Out-of-sample (test) | Evaluating the frozen configuration once | Tuning anything after the result is seen |
| Final holdout | One last evaluation of decisions already made | Any further selection |
Why the two curves diverge
Any optimiser fits both the repeatable structure of the training data and its accidental patterns. The accidental part does not recur, so the out-of-sample result keeps only the structural part. A steady in-sample climb followed by a flat or falling out-of-sample segment is the ordinary signature of a rule that mostly fitted noise. Some divergence is expected even for a sound rule; what matters is how much of the in-sample result survives, and whether the surviving part covers costs.
Ways to obtain out-of-sample evidence
- A single chronological split: train on the earlier part, test once on the later part.
- Walk-forward: repeated splits along the history, each test window preceded by its own training window.
- A different instrument or session for the same rule, when the rule's logic is not instrument-specific.
- Forward testing on a simulated account after the historical work, which adds live data feed behaviour but no fill certainty.
In Senzoukria
After a walk-forward run in the Automatic backtest panel, the report shows a chart labelled In-sample vs out-of-sample, with the curve the optimiser saw drawn dashed against the one you would actually have had, each over its own trade count. The panel's hint states the reading: a straight in-sample climb over a flat out-of-sample is what overfitting looks like, before any number says it.
Each slice in the report is listed with its Training period and its Real test period, so the boundary between the two periods is visible rather than implied. The Gauntlet section adds a Deflated Sharpe computed on the out-of-sample trades and the number of configurations tried. These reports summarise their own protocol on the supplied data; a favourable label is not a certification of future results.
Common mistakes
- Looking at the test period while designing the rule, then declaring it out-of-sample.
- Re-optimising on the holdout after a poor result, which quietly turns it into training data.
- Choosing a test period so short that a few trades decide the verdict.
- Reporting the in-sample curve in a summary and mentioning the out-of-sample curve only in the appendix.
Related
This page in other languages
Frequently asked questions
- How much data should be out-of-sample?
- There is no universal ratio. The test period needs enough trades for its result to mean something, and the training period needs enough to fit the rule without chasing single events. Decide the split from the trade frequency of the rule before running anything, write it down, and do not move it after seeing results.
- Can I reuse an out-of-sample period?
- Only at the cost of its status. Every decision taken after looking at a period's result makes that period part of the selection process. Reserve a final untouched period for the last evaluation and accept that once it has been used to decide something, a fresh period is needed for the next decision.
- Is a good out-of-sample result proof of an edge?
- It is better evidence than an in-sample result, not proof. The out-of-sample period is still one draw of history under one execution model and one set of costs. Sensitivity to higher costs, stability across neighbouring parameters and a record of how many rules were tried are needed to weigh it.