The gauntlet

The gauntlet is the heart of Holdout Lab: a deliberately strict, anti-overfit pipeline that decides whether a strategy has a real edge or just a lucky fit. A positive backtest is the start of the conversation, never the end. Each stage protects against a specific way to fool yourself.

You can run these stages individually from the Backtests tab (each is a run kind), as one command from the CLI (python -m ats.research), or unattended across the whole basket (python -m ats.sweep). Their combined verdict drives the readiness score on the research ladder.

The stages

1. Single backtest — the baseline

One historical simulation over the chosen window. Reads only the local catalog, fills on the next bar (no look-ahead). Produces the full metrics, the equity curve overlaid with buy-and-hold, a drawdown subplot, the round-trips, and a Monte Carlo panel. What it protects against: nothing yet — it’s the hypothesis, not the proof.

2. Rule test (vs random) — does the entry matter?

The strategy runs for real, then N times with its entry rule replaced by random entries at the same frequency (same exits, sizing, fees, data). If the real Sharpe doesn’t clear the 95th percentile of that random-entry cloud, the entry timing is adding little — the exits/holding are doing the work. Protects against: mistaking a generic holding-period effect for a real signal.

3. Optimize (Optuna) — with a train/test split

A seeded TPE search over the strategy’s search space, one backtest per trial. Set a train/test split date and the optimizer only ever sees the training window; the top-K trials each get a Monte Carlo verdict and held-out test metrics, and the pipeline recommends the best candidate that isn’t lucky. Protects against: picking parameters by peeking at the test set.

4. Walk-forward & WFO — does it survive out-of-sample?

Walk-forward runs the same parameters on N out-of-sample windows. WFO (walk-forward optimization) is stronger: parameters are re-optimized on each window using only the past, traded on the next window, and the out-of-sample steps are stitched into one honest equity curve. WFO’s detail view includes a parameter-stability table — wild swings window-to-window mean the optimizer is fitting noise. Protects against: a single in-sample fit that won’t generalize.

5. Monte Carlo — how lucky was the ordering?

Resamples the trade sequence to estimate the distribution of outcomes. A strategy whose result depends on one fortunate ordering of trades is fragile. Protects against: path luck.

6. Deflated Sharpe (DSR) — corrected for the search

The Sharpe ratio discounted for how many variants were tried (ats/workers/deflated_sharpe.py). Searching hard for a high Sharpe is mass multiple-testing; without this correction you crown noise. A DSR above ~0.95 is the hard gate — confidence the edge is real, not the luckiest trial. Protects against: selection bias from trying many strategies/parameters.

7. Cost stress — does the edge survive friction?

Re-scores the strategy at, e.g., 2× the modeled trading costs. Thin edges evaporate under realistic or doubled costs. Protects against: an edge that exists only at zero/under-modeled cost.

8. Buy-and-hold benchmark — is it worth the complexity?

Every run reports the instrument’s buy-and-hold return and the strategy’s excess over it. Beating buy-and-hold (net of cost and risk) is the bar that actually matters. Protects against: dressing up plain market exposure as alpha.

9. Cross-sectional breadth — does it show up on many names?

A real edge appears across many independent instruments, not two or three lucky ones. Breadth is a score bonus, never a gate, so a genuinely asset-specific edge (a structural pair, a sector anomaly) still qualifies — but a “winner” that works on only one name is treated with suspicion. See multi-instrument engines.

The hard gates

A strategy earns a ⭐ on the research ladder only when it clears all of:

1. the rule test is significant (beats the 95th percentile of random entry), and
2. held-out Sharpe > 0.5, and
3. walk-forward out-of-sample Sharpe > 0.5, and
4. deflated Sharpe > ~0.95.

These bars are deliberately strict: a positive rule test plus a positive held-out Sharpe is not enough. Monte Carlo and cross-sectional breadth are visible signals that inform the score but are not gates. Full scoring is on the research ladder.

Reading the verdict (in the app)

The Research tab doesn’t just show a red or green dot per rung — it can explain why. Open a strategy’s detail, and every rung on the ladder has a small “Why?” toggle: click it for a plain-language read of what that test checks, your actual result, and what the outcome means (e.g. “random entry days did about as well — the edge is in your exits, not your entry rule”). Where a concept has a lesson, a “Learn” link jumps straight to it in the Learn course. The Home tab’s strategy journey card and the “How the ladder works” panel carry the same links.

It’s entirely opt-in: nothing expands until you click, so the ladder stays a fast scan for anyone who just wants the numbers. Turning a rejection into a lesson is the point — the gauntlet is most useful when a failed strategy teaches you why, so the next one is better.

The recurring lesson

Across the built-in research, the gauntlet keeps teaching the same thing: simple edges win, and added machinery rarely adds rule-test significance. The honest record — what survived and what was refuted — is in the strategy library and the repo’s research notes.