v0.15.0: 26 of Our 27 Backtests Were Identical — and the Optimiser Ranked Them Anyway

The number that was not there

A user loaded a regime-switching mean-reversion model into our backtester — a hidden Markov model for the regime, a Hurst exponent for confirmation, fifteen settings of its own. He turned on the walk-forward, and the panel offered three parameters to sweep: lookback, stopTicks, targetTicks.

Those three are the settings of the example strategy we ship. They were written into the panel as a constant. His model reads exactly one of them.

So the optimiser ran 27 combinations across four folds. Twenty-six of them were bit-identical to each other, because sweeping a key a strategy never reads changes nothing — params.get("stopTicks", 40) returns 40 whether you passed a value or not. The ranking still sorted them. A winner still appeared. It was chosen by floating-point rounding.

No error was raised, because nothing went wrong. Every component did its job. The sweep expanded its axes correctly. The backtest computed correct statistics on correct bars. The sort was a correct sort. The defect lived one level up, in what the panel offered — and a measuring instrument that returns a confident number for a question nobody asked is worse than one that breaks.

Why the settings were invisible

The panel built its baseline from the strategy’s declared parameters — the list you fill in by hand in the editor panel. But a serious script reads its settings where it needs them, usually behind a one-line alias:

p = lambda key, default: float(params.get(key, default))
hurst_window = int(p("hurstWindow", 50))
band_k       = p("bandK", 2.75)

Fifteen settings in that model. Zero declared. The panel had no way to see them, so it fell back to a list written before the model existed.

What we shipped

v0.15.0 reads the source. It covers the forms people actually write — params.get(k, d), params[k], params.k ?? d — and it follows aliases, which is the part that matters: without alias support that model still yields zero parameters.

The axes it proposes now come from the script, with values around its own defaults. The step follows what the number is, not how big it is: an integer of 2 or more is a window, so it halves and doubles — ±1 on a window of 50 measures nothing. A number in ]0, 1] is a probability, so it moves by 0.1, bounded away from the edges. Anything else is a factor: ±25%.

Contract size is never proposed. It multiplies the result without changing a single decision, so sweeping it buys you the same SQN three times, in three times the runtime.

And a key the script does not read is now flagged. That was the most dangerous gap of all: a typo raises no error, it just sweeps nothing.

Two things we measured on the way

Sorting the parameters by where their name first appears in the file reads the comments. On our own default strategy the order came out stopTicks, targetTicks, fast — while the code reads fast first. The position has to be recorded at the match, not looked up afterwards.

And loading five years of one-minute bars killed the renderer outright: a white screen during “Loading bars…”, no log, no message. We rebuilt the exact payload from the real cache — 1,788,154 bars, 459 MB of JSON in a single block, of which 211 MB was actually used. The 139 bytes per row of difference were three strings the backtest never touches. It now arrives in pages, with a counter and a working cancel button.

The second half: a backtest that admits it is lying

Finding the sweep defect only sharpens the real question. If you try 400 configurations, the best one will look good. That is not evidence; it is arithmetic.

So v0.15.0 also ships the deflated Sharpe ratio (Bailey and López de Prado, 2014). It computes what the maximum of N independent trials would score by luck alone, corrects for the skew and fat tails of your actual return distribution, and reports the probability that your best result is not simply the expected maximum of that many coin flips.

We tested it the only way worth testing it: 400 draws of pure noise, no edge by construction, and asked it to judge the winner. It unmasks it.

The same release adds a regime-switching mean-reversion model to the script gallery, in Python — HMM for the regime, Hurst for confirmation. Its own header carries a measurement we think matters more than the model: on pure white noise, where the honest answer is 0.500, the classical rescaled-range estimator reads H = 0.630 at a window of 50. We wrote a whole post about that bias, and the model’s threshold is calibrated below what noise produces rather than below the textbook 0.5.

How this compares — and what we will not claim

The honest comparison here is not a feature table. We have not audited the internals of ATAS, Bookmap, Sierra Chart or Quantower, and we are not going to tell you what their optimisers do or do not do based on a marketing page. What we can offer is the test we now apply to our own, which you can run against any platform in ten minutes:

• Load a strategy whose settings are not declared in the UI — read from code, behind a helper. Does the optimiser find them, or does it offer you a fixed list?
• Sweep a parameter name that does not exist. Misspell it. Does anything object, or do you get a clean ranking of identical runs? Ours ranked them until this release.
• Ask what the reported Sharpe is being compared against. Zero, or the expected maximum of the number of configurations you tried? The gap between those two is where most backtested edges live.
• Load five years of one-minute bars. Watch what happens.

We failed the first two of our own tests until this release, and the fourth until last week. We are writing them down because a build log that only reports wins is an advertisement.

On the one comparison we have researched and keep dated — price — the picture is on our under-$30 comparison and the per-platform pages for ATAS, Bookmap, Sierra Chart and Quantower. Those pages carry verified prices and tier traps, and they get re-checked. This one carries measurements from our own instrument. We would rather keep the two separate than publish a table where half the cells are guesses.

The rest of v0.15.0

A seven-page GEX workspace that stops the endless scroll. A News terminal with a rotating globe built without a 3D dependency. A second AI engine with internet and read-only access to your own charts and backtests. Session-anchored cumulative delta and daily bars. MBO depth moved onto exchange time, matching the trades. And an autopilot that can run a scripted strategy on a live account — armed by hand, never persisted, with a daily loss limit and a contract cap.

Full notes ship with the release. The update reaches existing installs on next launch.