Operator runbook

For the team operating the service. Everything here is off by default or a plain HTTP call — nothing needs a customer to do anything.

Health probes

Endpoint	Auth	Use
GET /healthz	none	Liveness — “the process is up”. Cheap, dependency-free; orchestrator restart signal.
GET /readyz	none	Readiness — probes the database (and Redis when it’s the limiter backend). 503 drains the instance from the load balancer. Body carries per-check status only — the raw dependency error is logged, never returned (this probe is unauthenticated).
GET /status	none	Public status summary — coarse, marketing-named components (operational / degraded / down). Always 200 so a status page renders even during an outage. No internal detail.

A quota 402 is a healthy response and must not count against uptime — the structured status codes make an outage distinguishable from an expected refusal.

Scheduled jobs (cron)

Run as scheduled machines off the same image — see fly.toml. All are idempotent and safe to run aggressively.

python -m app.lifecycle    # daily  — trial expiry, past_due dunning reminders, purge worklist
python -m app.margin       # weekly — COGS vs billed margin telemetry per plan
python -m app.reconcile    # daily  — bill state drift: entitlements ↔ provider subscriptions
python -m app.overage      # daily  — billable overage per tenant → provider meter events

lifecycle and reconcile/overage only report or identify — they never delete data or repair billing silently. reconcile and overage are a clean no-op until a live billing provider is configured.

Billing operator views

Privileged, super-admin only:

• GET /admin/reconcile — drift between local entitlements and the billing provider’s subscriptions (detector, not repairer).
• GET /admin/overage — billable overage per tenant this cycle.
• GET /admin/licences/{org}/billing — one licence’s billing state: status, seat drift, trial/period dates, overage cap, cycle credits.
• GET /admin/metrics — activation, weekly-active seats/tenants, 7-day spend.

Tracing

OpenTelemetry traces on the metered path are off by default (a true no-op). Set PLTF_OTEL_EXPORTER_OTLP_ENDPOINT and install the otel extra to export spans; each AI call becomes one ai.metered_complete span (model, plan, decision code, credits — never prompt content).

Backups & recovery

Managed Postgres PITR plus a nightly pg_dump to a separate object-storage bucket with its own token. Targets and the restore drill are in the repo’s docs/RESTORE-RUNBOOK.md.