HTTP API reference (faucet serve)

faucet serve exposes a JSON REST control plane for submitting, polling, listing, cancelling, and streaming the logs of pipeline runs, plus unauthenticated health and Prometheus endpoints. A machine-readable docs/openapi.yaml spec ships alongside this page and is kept in sync with the router by a CI test.

See the serve cookbook for a guided quickstart, the security model, and operational guidance. This page is the endpoint reference.

Authentication

All /v1/* endpoints require Authorization: Bearer <token> unless the server was started with --no-auth. The token is compared in constant time; the Authorization header is the only accepted credential (no query-string auth). /healthz, /readyz, and /metrics are always unauthenticated (probes / scrapers). OPTIONS preflight bypasses auth so browsers behind a CORS policy work.

RBAC & the audit log (--auth-config)

A single --auth-token is one implicit admin principal. For a team deployment, --auth-config <file> promotes the server to role-based access control: a YAML/JSON file of principals, each a { name, token, role }. Three built-in roles form a ladder:

# auth.yaml
principals:
  - { name: alice, token: "${env:ALICE_TOKEN}", role: admin }
  - { name: ci,    token: "${env:CI_TOKEN}",    role: operator }
  - { name: dash,  token: "${env:DASH_TOKEN}",  role: viewer }

faucet serve --auth-config auth.yaml

A request whose role lacks the route’s required permission gets 403 forbidden (and a denied audit record). --auth-config is mutually exclusive with --auth-token / --no-auth. Every token is registered for log redaction at startup.

Audit log. Every mutating action (run.submit / run.cancel / run.delete) and every denied attempt is recorded with principal, role, action, run id, config fingerprint (submit), source IP, timestamp, and result. Admins read it via GET /v1/audit. Records persist in the run-history backend (faucet_serve_audit for the SQL backends; an in-memory ring otherwise) and expire with the --retain-terminal-runs-secs window.

Endpoints

POST /v1/runs

Request body:

{
  "config": "version: 1\npipeline:\n  source: {...}\n  sink: {...}\n",
  "config_format": "yaml",
  "name": "nightly-rollup",
  "labels": {"requester": "airflow"},
  "timeout_secs": 3600,
  "doctor_first": true,
  "idempotency_key": "airflow-task-123-attempt-2",
  "clock": "2026-05-29T00:00:00Z"
}

• config (required) — the YAML or JSON pipeline body.
• config_format — yaml (default) or json.
• name — metadata; also drives the state-key and metric identity (see the cookbook’s cardinality note). Two submissions sharing a name share replication bookmarks.
• labels — arbitrary string metadata, stored on the run record only.
• timeout_secs — wall-clock cap; on expiry the run is marked failed.
• doctor_first — run preflight probes before executing; on any failure the submit returns 422 with the doctor report in error.details.
• idempotency_key — replay protection (see cookbook).
• clock — overrides the ${now.*} clock for backfills (default: submit time).

Response (202):

{ "run_id": "0192…", "status": "queued", "submitted_at": "2026-05-29T12:00:00Z" }

A --default-config (if the server was started with one) is merged under the submitted config (submitted values win).

GET /v1/runs

Query parameters: status, name, since, until (RFC3339), limit (default 50, max 500), cursor. Ordering is (submitted_at DESC, run_id DESC); cursor is the last run_id from the previous page.

{ "runs": [ { "run_id": "…", "status": "completed", … } ], "next_cursor": "0192…" }

GET /v1/runs/{id} → RunRecord

{
  "run_id": "0192…",
  "name": "nightly-rollup",
  "labels": {"requester": "airflow"},
  "status": "completed",
  "submitted_at": "…", "started_at": "…", "finished_at": "…",
  "elapsed_secs": 12.4,
  "records_written": 4096,
  "invocations": [
    {"row_id": "default", "parent_record_key": null, "records_written": 4096, "error": null}
  ],
  "error": null,
  "idempotency_key": "airflow-task-123-attempt-2",
  "doctor_report": null
}

status is one of queued, running, completed, failed, cancelled. elapsed_secs is filled live for running runs.

Bookmarks: run records carry record counts + per-row outcomes, not replication bookmarks. Bookmark state is per-row/per-state-key and lives in the configured state backend, not in the run record.

GET /v1/runs/{id}/logs (SSE)

text/event-stream. The server replays the run’s bounded ring buffer, then streams the live tail. Event types:

• event: log — one captured log line (subject to the server’s FAUCET_LOG level; secrets are redacted).
• event: truncated — the reader fell behind and lines were dropped; rely on the centralized log sink for the full history.
• event: end — the run reached a terminal state; the stream closes.

Log buffers are ephemeral: they survive a short drain window after the run finishes (independent of run-record retention), then are dropped. A known run whose buffer has expired yields a single end.

curl -N -H "Authorization: Bearer $TOKEN" \
  http://127.0.0.1:8080/v1/runs/0192…/logs

GET /v1/catalog/* (Data Movement Catalog)

Read-only browsing of the Data Movement Catalog accumulated in the server’s --history backend (every serve run records into it automatically). Viewer-readable under RBAC; requires a build with the catalog feature.

• GET /v1/catalog/datasets?kind=&q=&limit=&cursor= — paginated dataset list, ordered (last_seen DESC, id DESC); q is a case-insensitive URI substring.
• GET /v1/catalog/datasets/{id} — the dataset plus its deduplicated schema timeline (each version with a diff vs the previous), recent per-run volume points, and upstream/downstream lineage edges. 404 for an unknown id.
• GET /v1/catalog/lineage?root=&depth= — the source→sink edge graph; with root (a dataset id), a BFS slice bounded by depth hops.

curl -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8080/v1/catalog/datasets?kind=postgres&limit=20"

POST /v1/backfill

Plans a [from, to) range into window units (chunked by window) and submits one tracked run per unit — see the backfill cookbook for the model.

{
  "config": "version: 1\nname: orders\npipeline: {...}\n",
  "config_format": "yaml",
  "from": "2026-06-01",
  "to": "2026-07-01",
  "window": "1d",
  "timezone": "UTC",
  "name": "orders",
  "labels": {"requester": "airflow"},
  "timeout_secs": 3600
}

• config (required) — every root source must reference a ${backfill.*} or ${now.*} scoping token (400 otherwise). Bookmark-range backfills are CLI-only.
• from / to (required) — RFC3339 or YYYY-MM-DD (midnight in timezone), half-open.
• window / timezone — default to the config’s backfill: block.
• name — base run name; unit runs are {name}-backfill-{unit} (the pipeline name is rewritten per unit so state keys never touch the live bookmark). delivery is forced to at_least_once; timeout_secs applies per unit.

202 response: {backfill, descriptor, planned, submitted, units: [{unit, start, end, status, run_id?, error?}]} where backfill is the stable range hash carried as the backfill label on every unit run (plus a backfill_unit label). Each unit is submitted with the deterministic idempotency key backfill:{hash}:{unit}, so re-POSTing the same body is replay-safe — already-submitted units replay their existing run, the rest submit (a full queue marks the remainder not_submitted; re-POST to continue). A config carrying shard: {count} makes each unit a sharded run tracked via shard progress. Requires RunWrite (operator); audited as backfill.submit.

Error envelope

Every error is a JSON ApiError:

{ "error": { "code": "unprocessable", "message": "…", "details": { } } }

Metrics

/metrics serves the standard faucet_* pipeline metrics plus serve-specific series: faucet_serve_requests_total{method,path,status}, faucet_serve_request_duration_seconds{method,path}, faucet_serve_runs_queued, faucet_serve_runs_in_flight, faucet_serve_runs_total{status,reason}, faucet_serve_idempotency_hits_total, and faucet_serve_history_degraded. See Observability.