Running in production (always-on)

The deployment doc covers how the processes run. This one covers keeping them running unattended: process supervision that survives crashes and reboots, a deep health probe, scheduled backups with a tested restore, and log hygiene. This is what turns the dev-mode scripts/start.sh into a machine you can walk away from while a live campaign runs.

Everything here targets the supported host: macOS with launchd. There is no systemd/Linux packaging — the Docker app profile stays evaluation-only.

The shape

Tier	Supervised by	Restarts on
Postgres, Redis	Docker Desktop (“Start at login”)	login, crash
API, worker, frontend	launchd LaunchAgents (com.ats.{api,worker,frontend})	login, crash
Paper/live runners	the API (paper_supervisor)	gateway heal, manual
Daily housekeeping	launchd (com.ats.maintenance)	its schedule

launchd supervises the app tier; the API supervises the trading runners. The runners deliberately stay under the API so a worker restart never orphans an open position.

1. Production start mode

scripts/start.sh --prod

vs. the default dev mode this drops uvicorn’s --reload (no code-watcher restarting mid-trade) and serves a built frontend (npm run build → vite preview on a pinned :5173) instead of the Vite dev server. Dev mode (scripts/start.sh, no flag) is unchanged.

scripts/stop.sh SIGTERMs each process and waits up to 20s before SIGKILL — the worker needs that window to finish its in-flight arq job and let the relay flush its final session snapshot. --keep-services leaves Postgres/Redis up.

2. launchd supervision

Prerequisite: Docker Desktop → Settings → General → Start Docker Desktop when you sign in. The agents wait for the daemon but do not launch it.

scripts/install-launchd.sh            # render templates → ~/Library/LaunchAgents, bootstrap, start
scripts/install-launchd.sh --status   # state / pid / last exit code per agent
scripts/install-launchd.sh --uninstall # bootout + remove

The three app agents carry KeepAlive={SuccessfulExit:false} (restart on crash, not on a clean stop), RunAtLoad (start at login), ThrottleInterval=15, and a PATH widened to reach uv, docker, and node (launchd’s default PATH has none of them). Each runs a wrapper (scripts/run-*.sh) that waits for Docker, brings up Postgres+Redis, applies migrations (API only), then execs the real process so launchd supervises it directly.

Because KeepAlive would respawn a process you just killed, scripts/stop.sh boots out any loaded agents first. To bring the app tier back after a stop.sh, re-run scripts/install-launchd.sh.

Verify supervision actually works:

scripts/install-launchd.sh --status                       # all three "running"
kill -9 "$(cat .run/api.pid 2>/dev/null || pgrep -f 'uvicorn ats.api')"
sleep 20 && curl -fsS localhost:8000/api/health           # KeepAlive brought it back

3. Deep health probe

GET /api/health stays a bare liveness ping (the Docker healthcheck depends on its shape). GET /api/health/deep reports each dependency and is what an uptime monitor should watch:

curl -s localhost:8000/api/health/deep | jq
# { "ok": true, "components": {
#     "db": {"ok": true, ...}, "redis": {"ok": true, ...},
#     "worker": {"ok": true, "detail": "...j_complete=..."},   # arq heartbeat, refreshed every 60s
#     "catalog": {"ok": true, ...} } }

It is always HTTP 200 (a dead Redis flips redis.ok to false rather than turning the probe into a 500 that hides which component failed) and unauthenticated (probes can’t log in; the body is operational state only). worker.ok goes false within ~60s of the worker dying.

4. Backups

scripts/backup.sh                   # Postgres dump + catalog mirror + .env
scripts/backup.sh --with-artifacts  # also the ~4GB backtest-artifacts corpus (weekly/manual)

• Destination is $ATS_BACKUP_DIR (default ~/ATS-backups). Point it at an external volume — a backup on the same disk survives a fat-finger, not a disk failure. The com.ats.maintenance agent sets ATS_BACKUP_DIR; edit its plist to relocate.
• What’s backed up: the Postgres DB (pg_dump | gzip), the market-data catalog (rsync backend/data, grow-only — no --delete, so a local deletion/corruption can’t propagate to the backup of the billed Databento data), and .env (chmod 600, versioned — a dated env-<stamp>.backup plus the env.backup latest copy). Artifacts are large and reproducible from the DB + catalog, so they’re opt-in.
• Retention: newest 7 daily DB dumps + 4 weekly (Sunday) dumps + 7 dated .env copies; the catalog is a current mirror.

Run it three ways: the daily com.ats.maintenance agent (below); Settings → System → Back up now (the one-click button); or scripts/backup.sh by hand. All are safe anytime — pg_dump is a consistent snapshot, so no need to stop trading.

5. Log rotation + cleanup

com.ats.maintenance runs scripts/maintenance.sh daily at 03:45 local (07:45–08:45 UTC across US DST — always before the 09:30 UTC fidelity cron). It:

1. rotates .run/*.log over 50MB using copytruncate (copy then truncate in place — a rename would leave launchd’s log fd pointed at the moved file; keeps the newest 5 compressed rotations),
2. deletes per-session paper logs (backend/artifacts/paper-logs) untouched for 30+ days (an active session writes continuously, so a stale file is always a finished one),
3. takes a scripts/backup.sh.

6. Restore

Recover for real — scripts/restore.sh

The symmetric inverse of backup.sh, in one command. It restores the DB, catalog, and .env together and is CLI-only — restore overwrites all live state, so it is never exposed as a browser button.

scripts/stop.sh          # stop the app first (no connections competing with the schema reset)
scripts/restore.sh       # newest daily dump; prompts for a typed RESTORE confirmation
scripts/start.sh         # back up

# variants
scripts/restore.sh --dump "$ATS_BACKUP_DIR/db/daily/ats-<stamp>.sql.gz"  # a specific dump
scripts/restore.sh --skip-env    # keep the current backend/.env (don't restore keys)
scripts/restore.sh --yes         # skip the interactive confirm (automation)
scripts/restore.sh --force       # proceed even if the API is still serving

It refuses if a paper/live runner is active (ats.paper / ats.crypto_paper / ats.live), and (without --force) if the API is still up. It then resets the schema, restores the dump, rsyncs the catalog back into backend/data, restores the newest .env, and runs alembic upgrade head (a no-op if the dump is already at head — the guard against a dump taken mid-migration).

Verify a backup without touching live — throwaway container

A backup you have never restored is a guess. To rehearse safely, restore into a scratch Postgres:

BK="$ATS_BACKUP_DIR/db/daily"                 # or wherever the dump lives
DUMP="$(ls -1t "$BK"/ats-*.sql.gz | head -1)" # newest dump

docker run -d --name ats-restore-test -e POSTGRES_USER=ats \
  -e POSTGRES_PASSWORD=ats -e POSTGRES_DB=ats -p 5544:5432 postgres:17
sleep 5
gzip -dc "$DUMP" | docker exec -i ats-restore-test psql -U ats -d ats
docker exec ats-restore-test psql -U ats -d ats -c \
  "SELECT count(*) FROM session_snapshots; SELECT count(*) FROM trading_sessions;"
docker rm -f ats-restore-test

Non-goals

No systemd/Linux packaging (macOS launchd is the prod target). No 2FA automation for IB re-login (it physically requires you). Paper/live runners stay under paper_supervisor, not launchd — the API owns their lifecycle so a restart never orphans a position.