Skip to content
ClearEdge AlgoClearEdge Algo

Data ingestion

Ingestion pulls vendor data into the local catalog. You drive it from the Data tab (or the CLI), pick a source, and the work runs as a background job with a live progress bar and ETA. All sources use bring-your-own keys (see configuration).

Sources

Section titled “Sources”

Tiingo — daily (free)

Section titled “Tiingo — daily (free)”

Daily OHLCV for equities/ETFs and crypto. Needs ATS_TIINGO_API_TOKEN (free). This is the default starter source; daily history is small and free, so ingest generously.

Databento — intraday 1-minute (billed)

Section titled “Databento — intraday 1-minute (billed)”

Intraday strategies need 1-minute bars. Databento ingestion (ats/data/databento_client.py, ats/data/intraday_ingest.py) pulls raw trades and builds 1-minute bars plus a VWAP series and a measured spread sidecar (used to charge realistic intraday costs in backtests). Needs ATS_DATABENTO_API_KEY. Because Databento bills per request, the Data tab shows a cost-estimate confirm step before it runs, and the system only fetches data you don’t already have — so re-ingesting an overlapping range is cheap.

From the command line, ats.intraday does the same intraday pull with the same cost preview:

Terminal window
cd backend
uv run python -m ats.intraday --symbols SPY,QQQ,IWM --start 2023-03-28 --end 2026-06-13  # full pull
uv run python -m ats.intraday --symbols SPY --start 2026-06-12 --end 2026-06-13 --estimate-only

--estimate-only prints the Databento cost and stops — the terminal equivalent of the Data tab’s confirm step.

Crypto (Tiingo crypto)

Section titled “Crypto (Tiingo crypto)”

Crypto daily (and 1-minute) bars via Tiingo’s crypto endpoints (ats/data/crypto_ingest.py), written as …​.CRYPTO instruments. Crypto 1-minute history is fetched in yearly chunks to respect rate limits.

The Data tab

Section titled “The Data tab”
  • Source picker — Tiingo daily / Databento intraday / crypto.
  • Single-symbol form — symbol, date range, and (for Databento) the cost estimate.
  • Bulk ingest — ingest a basket of liquid ETFs in one action (the validated research universe).
  • Progress + ETA — a live progress bar with an estimated time remaining, published from the worker to Redis (ats/data/ingest_progress.py) and streamed to the UI.
  • Per-row storage size — each catalog row shows its on-disk size, so you can see what intraday data costs you in space.

How it flows

Section titled “How it flows”
  1. You submit an ingest from the Data tab → an arq job is enqueued.
  2. The worker fetches from the vendor into ATS_RAW_DATA_DIR, builds bars (and VWAP/spread for intraday), and writes them into the catalog, publishing progress to Redis as it goes.
  3. The new instrument/range appears in the Data tab and is immediately usable in backtests and charts.
Section titled “Related”