Respect the holdout.
A self-hosted lab with the whole path in one app — a hands-on course, an AI strategy studio, a nine-stage anti-overfit gauntlet, and paper trading — so you find out what actually holds up before a dollar is at risk.
Runs on your machine · your data · your broker · no real money at stake
Everyone getting into algo trading hits the same wall
Every tool draws a gorgeous equity curve. None of them warn you it's overfit.
The tutorials stop at a backtest. The very next step is real money.
Signal-sellers and curve-fit gurus everywhere. The truth, nowhere.
Holdout Lab is built to tell you the truth — and that's exactly what a newcomer needs to learn.
Typical backtester · in-sample
“+312% · Sharpe 2.8 · looks amazing!”
Holdout Lab verdict · out-of-sample
“Overfit. Out-of-sample edge ≈ 0. Don’t trade this.”
The whole lab, tab by tab
Holdout Lab is a single self-hosted app. Here's what each tab does — honestly.
Start with a course that runs real backtests, then build with AI, your own data, and real charts.
Four modules take you from foundations to the overfitting trap to the gauntlet — and every lesson has you doing it, not reading about it.
The Studio turns a sentence into a working NautilusTrader strategy with your own Anthropic key — then validates and sandboxes it before it touches a single bar.
Free daily equities, free 1-minute crypto, and pay-per-use intraday data — every pull lands in a local Parquet catalog on your machine.
Saved chart workspaces with drawings and indicators — and any strategy's entries and exits overlaid on the candles it actually saw.
Backtest every idea, then let the honest scoreboard tell you which ones actually earned trust.
Full metrics, equity vs buy-and-hold, drawdown, and a Monte Carlo band — with no look-ahead by construction: closed bars, next-bar fills, modeled costs.
Every strategy × symbol × timeframe you research is scored on one board — with the four hard gates it must clear shown right next to the number.
Forward-test with zero money at risk — on rails built like real money is on the line.
The exact strategy object you backtested runs against live market data with no money at stake — the honest dress rehearsal most tools skip.
The live path exists so the discipline is real: every order must clear a pre-trade risk engine before the broker ever sees it. The free edition ships research and paper only — live trading stays edition-gated.
Every idea runs the gauntlet
Every stage exists because of a specific way people lose money to their own numbers.
Walk-forward optimization re-tunes the strategy on each past window and trades the next one out-of-sample, stitching the honest segments into one curve. A parameter-stability check flags strategies that are just fitting noise.
A single equity curve is one roll of the dice. Holdout Lab runs a Monte Carlo over thousands of resampled paths, so you see the median and the 5–95% band — the realistic spread, not a cherry-picked best case.
How you run it
Download the bundle, run two commands, open your browser. Single-user auth, nothing hosted by us, no cloud account.
Tiingo, Databento, and Anthropic keys live in your local settings — with a Test button for each. They never touch our servers, because we don't have any.
The code that backtests is the code that trades: the same NautilusTrader strategy object runs in backtest, paper, and live. No reimplementation gap.
Questions
No. Describe a strategy in plain English and the Studio writes it for you, or start from a validated template. If you want full control, you can write Python — but you never have to.
Yes. Holdout Lab is self-hosted — it runs on your own machine. Your keys, strategies, and trades never leave your computer and are never sent to us.
US stocks & ETFs and crypto. Paper trading runs against Interactive Brokers (paper account) or the built-in zero-setup crypto simulator. Data comes from Tiingo (daily equities), Binance (1-minute crypto), and Databento (intraday equities).
The engine has a complete live-trading path — pre-trade risk limits, an audit-logged approve-to-live gate, a kill switch — but the free edition ships research and paper trading only. Live trading stays edition-gated.
It is sandboxed — no file, network, or system access — statically validated, and run in an isolated process. Nothing can ever go live until it clears the gauntlet and you explicitly approve it.
No. It runs locally on your own machine (via Docker). Nothing is hosted by us, and there is no cloud account to manage.
A Mac or Linux machine with Docker and about fifteen minutes. One command brings the whole app up locally.
Tiingo daily US equities and Binance 1-minute crypto are free (Tiingo needs a free key; Binance needs none). Databento intraday data is pay-per-use — the app shows a cost estimate you must confirm before anything is billed.
The anti-overfit validation pipeline — nine stages, including a rule test against random-entry twins, optimization judged on held-out data, walk-forward, Monte Carlo, deflated Sharpe, a 2× cost stress, and a buy-and-hold benchmark. Only strategies that survive every stage are marked validated.
Backtests are simulations on historical data — simulated and past performance never guarantee future results. The whole point of Holdout Lab is honest validation, not flattering numbers.
The research + paper-trading edition is free — download it and run it on your own machine. A paid Pro edition with live-trading connectivity is planned; leave your email on this page to hear when it opens and what it costs.
The research + paper-trading edition is free and self-hosted. Download the bundle and you're backtesting in minutes.
A paid Pro edition with live-trading connectivity is planned. Leave your email to hear when it opens — and what it costs.
By joining, you agree to our Privacy Policy.