Data ingestion

Ingestion pulls vendor data into the local catalog. You drive it from the Data tab (or the CLI), pick a source, and the work runs as a background job with a live progress bar and ETA. All sources use bring-your-own keys (see configuration).

Sources

Tiingo — daily equities (free)

Daily OHLCV for equities/ETFs. Needs ATS_TIINGO_API_TOKEN (free). This is the default starter source for stocks; daily history is small and free, so ingest generously. (Crypto uses Binance — below.)

Databento — intraday 1-minute (billed)

Intraday strategies need 1-minute bars. Databento ingestion (ats/data/databento_client.py, ats/data/intraday_ingest.py) pulls raw trades and builds 1-minute bars plus a VWAP series and a measured spread sidecar (used to charge realistic intraday costs in backtests). Needs ATS_DATABENTO_API_KEY. Because Databento bills per request, the Data tab shows a cost-estimate confirm step before it runs, and the system only fetches data you don’t already have — so re-ingesting an overlapping range is cheap.

From the command line, ats.intraday does the same intraday pull with the same cost preview:

cd backend
uv run python -m ats.intraday --symbols SPY,QQQ,IWM --start 2023-03-28 --end 2026-06-13  # full pull
uv run python -m ats.intraday --symbols SPY --start 2026-06-12 --end 2026-06-13 --estimate-only

--estimate-only prints the Databento cost and stops — the terminal equivalent of the Data tab’s confirm step.

Crypto — Binance public klines (free)

Crypto bars come from Binance’s public data CDN (data.binance.vision) — free, no API key, no rate limit, no row cap, with history back to 2017 (ats/data/binance_klines.py). Monthly ZIPs for complete months plus daily ZIPs for the current month are downloaded, checksum-verified, parsed, and written as ….CRYPTO instruments via the shared writer in ats/data/crypto_ingest.py. Ingest 1-min once and the resampler derives every higher cadence for free (see resampling).

Symbols use the USD name the rest of ATS knows (BTCUSD, ETHUSD, …) and map to Binance’s USDT pairs under the hood (USDT ≈ $1). Two file-format quirks are handled automatically: Binance switched kline timestamps from milliseconds to microseconds on 2025-01-01, and some vintages include a header row.

Symbol formats

Getting the ticker right is the one thing that used to bite: a wrong symbol enqueued a job blind and failed with a dead-end “check the worker logs”. Now the Data tab validates the symbol live before you submit — a green ✓ with the instrument’s name and available date range, or a red ✗ with the reason — and normalizes it so one asset never gets two catalog identities.

Equities/ETFs — a bare US ticker: SPY, AAPL, QQQ. Share classes are hyphenated (BRK-B, not BRK.B) — that’s Tiingo’s form. You can type the dot; it’s normalized to the hyphen automatically, and the green check shows the symbol it will actually store.
Crypto — the canonical <BASE>USD pair: BTCUSD, ETHUSD, SOLUSD. The equivalent ways to write a USD pair — BTC, BTCUSD, BTCUSDT, BTCUSDC — all collapse to BTCUSD, so BTC only ever has one BTCUSD.CRYPTO catalog entry. (Binance still resolves the download to its USDT pair.)
Live check — equities are confirmed against Tiingo’s free, non-billed metadata endpoint (it never costs a request); crypto has no cheap metadata, so canonicalization is the check. When the check reports the vendor’s earliest bar and the From date is still the default, it’s prefilled for you.
Legible failures — if an ingest still fails (a delisted ticker, a vendor outage), the Data tab shows the vendor’s actual reason (“Tiingo returned no data for FOOBRA”), not a generic message.

The Data tab

Source picker — Tiingo daily (equities) / Databento intraday (equities) / Binance (crypto).
Single-symbol form — symbol (validated live, see Symbol formats), date range, and (for Databento) the cost estimate.
Bulk ingest — one-click starter baskets: equities (Tiingo daily) or crypto (Binance 1-min, full history — the resampler then covers daily + every intraday cadence). Crypto needs no API key.
Progress + ETA — a live progress bar with an estimated time remaining, published from the worker to Redis (ats/data/ingest_progress.py) and streamed to the UI.
Per-row storage size — each catalog row shows its on-disk size, so you can see what intraday data costs you in space.

Nightly refresh (automatic)

Once a dataset is in the catalog, you don’t have to keep it current by hand. A worker cron (08:45 UTC, after Binance publishes its daily CDN files and Tiingo’s end-of-day data lands) walks the whole catalog and tops up every free-vendor dataset to yesterday’s close:

Equity daily → Tiingo (needs ATS_TIINGO_API_TOKEN; skipped with a note when unset).
Crypto daily and 1-minute → Binance public klines (no key needed). After a 1-minute refresh, the cached resampled cadences (5-min, 1-hour, …) are invalidated so they rebuild from the fresh series on next use.
Resample-derived cadences are skipped — they rebuild on demand from 1-minute data.
Equity intraday is never touched. Databento is billed per request, so it is only ever pulled through the Data tab’s explicit cost-confirmed flow — never by a background job.

The refresh is incremental (it pulls only the missing tail after the last stored bar) and idempotent, and one failing symbol never aborts the rest. Set ATS_NIGHTLY_CATALOG_REFRESH=false to disable it.

How it flows

You submit an ingest from the Data tab → an arq job is enqueued.
The worker fetches from the vendor into ATS_RAW_DATA_DIR, builds bars (and VWAP/spread for intraday), and writes them into the catalog, publishing progress to Redis as it goes.
The new instrument/range appears in the Data tab and is immediately usable in backtests and charts.

The data catalog — where ingested data lives.
Configuration — the data keys.
Command-line research — bulk ingest from the CLI.