CLI commands

The faucet binary exposes these commands. Pass --log-level <level> (or set FAUCET_LOG) to control logging.

[config] is optional for run / validate / preview / doctor / replicate / schedule: if omitted, faucet auto-discovers faucet.yaml → .yml → .json in the current directory.

run

faucet run pipeline.yaml
faucet run                              # auto-discover faucet.yaml in cwd
faucet run --from-env                   # build the pipeline entirely from FAUCET_* env vars
faucet run pipeline.yaml --env-file prod.env
faucet run pipeline.yaml --no-env-file
faucet run pipeline.yaml --clock 2026-03-01          # backfill: set ${now.*} clock to midnight UTC
faucet run pipeline.yaml --clock 2026-03-01T02:00:00-08:00  # backfill: precise RFC 3339 timestamp

Flags:

validate

Reports one line per expanded matrix row. Use it in CI to catch config errors before deploying.

faucet validate pipeline.yaml

When the config contains secrets-manager directives (${vault:…}, ${aws-sm:…}, etc.), faucet validate resolves them as a real preflight and prints one confirmation line per reference (never the value):

secret: vault:secret/data/faucet/api#token → resolved
ok: 'my-pipeline' rows=1 (roots=1, children=0) execution=(defaults)
  - default [root] source=rest sink=jsonl delivery=at-least-once

Each row line ends with the derived end-to-end delivery guarantee for that row’s source × sink × config — at-least-once, effectively-once (atomic watermark), or effectively-once (keyed upsert) — computed regardless of the requested delivery: mode, so an upsert-keyed row is reported as effectively-once even without delivery: exactly_once. See delivery.

Pass --no-secrets to validate grammar and structure only, skipping all secret fetches. This is useful in CI environments that lack credentials, or in local development before vault access is available:

faucet validate --no-secrets pipeline.yaml

Composition flags

When a config uses composition (extends: / profiles: / !include), validate resolves it like run does:

faucet validate app.yaml --profile prod        # select a named overlay
faucet validate app.yaml --show-composed       # print the fully merged config

• --profile <name> selects a named overlay from profiles: (also settable via FAUCET_PROFILE; the flag wins). An undeclared name is a clear load-time error.
• --show-composed prints the fully composed document — bases merged, the selected profile applied, !include fragments substituted, and the extends: / profiles: metadata stripped — before ${...} interpolation. It’s the fastest way to confirm a multi-file setup resolves to what you expect.

discover

faucet discover conn.yaml                      # print a generated config to stdout
faucet discover conn.yaml -o pipeline.yaml     # write it to a file (--force to overwrite)
faucet discover conn.yaml --include 'public.*' --exclude '*.tmp_*'
faucet discover conn.yaml --source warehouse   # introspect a named pipeline.sources template
faucet discover conn.yaml --json               # machine-readable dataset list

Connects to the config’s source, enumerates the datasets behind it (tables / collections / indices / object-store prefixes), and emits a ready-to-run config with one matrix row per dataset — the input document with its matrix: block replaced, secrets echoed as raw ${…} references. The generated config passes faucet validate. Supported sources: postgres, mysql, mssql, sqlite, mongodb, elasticsearch, bigquery, snowflake, s3, gcs.

See the source discovery cookbook.

preview

Runs the first root row’s source and prints records (via the stdout sink). Children aren’t previewed because they need parent records to resolve ${parent.path} tokens.

faucet preview pipeline.yaml --limit 10
faucet preview app.yaml --profile dev --limit 5   # preview with a named profile overlay

--profile <name> / FAUCET_PROFILE selects a named overlay from profiles: before previewing. Same semantics as run and validate.

plan

A read-only “what would this config change” preview — it runs the sink’s non-mutating check() probe and pure schema/lineage analysis but never writes to any sink.

faucet plan pipeline.yaml
faucet plan pipeline.yaml --sample fixtures.jsonl        # preview output schema/volume offline
faucet plan pipeline.yaml --live --limit 20 --json       # capped read-only source pull, JSON out

Reports, for the selected row (--row, default the first root): the resolved source/sink/write-mode/delivery guarantee, the transform chain in lifecycle order, which quality/contract/masking/drift policies are in effect, and the lineage column ops. Given a sample (--sample <fixture> offline, or --live --limit N for a capped read-only source pull), it also reports the inferred output schema, the sink schema delta (adds / widenings / incompatible via diff_schema when the sink exposes current_schema(); “schemaless — no delta” otherwise), and a volume estimate. The data pass runs through the offline harness, so no sink is ever written. Offline by default; --resolve-secrets opts into the real secrets path.

dev

A watch-and-diff authoring loop (requires the cli-dev build feature). Re-runs a sample through the offline harness on every config save and prints the schema, DLQ count, errors, and a diff vs the previous run.

faucet dev pipeline.yaml --sample fixtures.jsonl

Watches the config file’s directory and the directories of any extends: / !include fragments, so editing an included fragment re-triggers a run. In a non-TTY (CI) or with --once it runs a single pass and exits. Debounce the watcher with --debounce-ms.

schema

faucet schema config          # the WHOLE config document (top-level grammar)
faucet schema source rest
faucet schema sink bigquery
faucet schema transform keys_case
faucet schema dlq
faucet schema execution
faucet schema contract
faucet schema masking
faucet schema sla
faucet schema notifications
faucet schema secrets
faucet schema triggers
faucet schema catalog

faucet schema config prints a composed JSON Schema for the entire faucet.yaml / faucet.json document — the top-level grammar (version, name, vars, auth, pipeline, matrix, execution, and every optional block such as schedule / lineage / quality / dlq / resilience that is compiled into your binary) plus per-connector type discrimination: the source / sink positions become a oneOf over the connector kinds your binary knows, each branch embedding that connector’s own config schema. Point an editor at it for autocomplete and validation as you type — see Editor setup.

faucet schema transform <name> prints the inline config schema for a transform (e.g. keys_case lists the valid mode: values). Run faucet list to see which transforms are compiled into your binary.

faucet schema execution prints the schema for the top-level execution: block, including concurrency, error handling, and adaptive batch sizing.

faucet schema masking prints the JSON Schema for the pipeline.masking: (PII detection + column-masking) block — see masking.

faucet schema sla prints the schema for the top-level sla: (freshness/volume SLA) block — see SLA monitoring.

faucet schema secrets prints the directive grammar and auth requirements for all four secrets-manager backends in machine-readable JSON — useful for tooling that needs to understand the interpolation syntax without reading the docs.

faucet schema triggers prints the JSON Schema for the --triggers file format (the TriggersFile / TriggerSpec / TriggerKind types). Requires the triggers Cargo feature.

faucet schema catalog prints the JSON Schema for the top-level catalog: (Data Movement Catalog store) block — see the catalog cookbook. Requires the catalog Cargo feature.

init

faucet init my_pipeline --source postgres --sink bigquery

Required fields are surfaced with a typed placeholder and a # REQUIRED marker; optional fields are commented out so connector defaults apply. The interactive mode (--interactive) is gated behind the cli-interactive feature.

Singer discovery. For the Singer bridge source, add --discover --executable <tap> to run the tap’s --discover, write the returned catalog to catalog.json, and scaffold a config that inlines the catalog and lists the discovered streams (with stream: left empty for you to choose):

faucet init --source singer --discover --executable tap-github -o pipeline.yaml

faucet doctor then verifies the tap resolves on PATH and that the selected stream exists in the catalog.

new

Scaffold a new connector crate (not a config) that follows every repo convention — ready to cargo build and publish:

faucet new connector acme --kind source            # → faucet-source-acme/
faucet new connector acme --kind sink --common      # + a faucet-common-acme/ crate
faucet new connector acme --kind source -o crates/  # write into crates/

The generated crate has the standard module layout (config.rs, stream.rs or sink.rs), a JsonSchema-deriving config, config_schema() / connector_name() overrides, the #![cfg_attr(docsrs, feature(doc_cfg))] crate-root line, the [package.metadata.docs.rs] block, system-name-first crates.io keywords, a README, and a passing unit test — so cargo test is green out of the box with a trivial passthrough. Fill in the TODOs, then publish. See Authoring a connector.

search / install / list --available

Discover connectors from the connector registry — a curated, feature-independent index of every built-in connector plus community faucet-source-* / faucet-sink-* crates.

faucet search kafka              # matches on name, description, keywords, crate
faucet search cdc --json         # machine-readable
faucet list --available          # the whole registry; ● = in this binary, ○ = installable
faucet install bigquery --kind sink
faucet install my-connector --index ./my-registry.json

faucet install <name> never runs anything — it prints the recipe:

• a built-in already compiled in → “already available”;
• a built-in not compiled in → cargo install faucet-cli --features <kind>-<name>;
• a community connector → a copy-pasteable custom-binary snippet (see Custom binaries).

--index <path> points any of these at a custom/mirror index instead of the built-in one. Ambiguous names (a connector that is both a source and a sink, e.g. postgres) need --kind source|sink.

conformance

Score every compiled-in connector against the faucet SDK contract and print its maturity tier — 🟢 Stable, 🟡 Experimental, 🟠 Beta, ⚪ Draft — plus its capability badges (exactly-once, discover, upsert, schema-evolution).

faucet conformance                 # score every connector, highest first
faucet conformance --kind sink     # sinks only
faucet conformance postgres        # a detailed scorecard for one connector
faucet conformance --json          # machine-readable scorecards

The score (0–100) is computed from authoritative, instantiation-free signals: a verified cli/connectors/registry.json entry (40) + a real config schema (30) form the Stable gate at 70; documentation, exactly-once delivery, and the kind-specific capability (source discovery / sink upsert + schema evolution) are bonuses on top. Every conforming built-in is Stable with capability badges; an incomplete third-party connector (missing a verified entry or a schema) lands at Experimental / Beta.

doctor

faucet doctor pipeline.yaml                  # checklist; exit code = # of failed probes
faucet doctor pipeline.yaml --timeout-secs 5 # per-probe timeout (default 10)
faucet doctor pipeline.yaml --json           # machine-readable, for CI gating
faucet doctor app.yaml --profile prod        # probe with a named profile overlay applied

Runs a fast, non-mutating preflight against every connector in the config so misconfiguration surfaces before a real run. For each root invocation it probes the source, sink, and state store and prints a green/red checklist with elapsed times; the exit code equals the number of failed probes (clamped to 255).

• Sources reuse the real read path — the probe pulls a single page and stops (never the full dataset). Sources whose first page would block or mutate use a targeted probe instead: webhook (port bindable), websocket (TCP connect), postgres-cdc (slot reachable), kafka (cluster metadata).
• Sinks run a read-only connect/auth/metadata call — SELECT 1, HeadBucket, PING, tables.get, cluster health, fetch_metadata, or a directory-writable check for file sinks. Never a real write.
• State stores do a sentinel put/get/delete that leaves no residue.
• SLA (when a top-level sla: block is configured) reads the persisted run history and reports staleness of the last successful run vs max_staleness_secs and volume-baseline warm-up state — read-only.

Child invocations (parent/child matrix rows) are listed but not probed — their configs depend on parent records that only exist at run time. Probe messages are scrubbed for resolved secrets before printing.

--profile <name> / FAUCET_PROFILE selects a named overlay from profiles: before probing (same semantics as run and validate).

See the Troubleshooting cookbook page for reading the output and common failures.

test

faucet test tests/*.yaml                    # run every case; exit code = # of failed cases
faucet test tests/orders.yaml --filter null # only cases whose name contains "null"
faucet test tests/*.yaml --json             # machine-readable { total, passed, failed, tests }
faucet test tests/*.yaml --clock 2026-03-01 # default ${now.*} clock for cases without clock:

Runs fixture-based, fully-offline pipeline tests. Each case in a spec file feeds sample records through the real transform → quality → contract path with an in-memory source, sink, and DLQ — the configured source and sink are never built or contacted — and asserts the output records, DLQ routing, counts, or an expected failure. The exit code equals the number of failed cases (clamped to 255), so CI gates on it directly.

Flags:

faucet schema test prints the spec file’s JSON Schema. See the Testing pipelines cookbook page for the spec grammar, matching semantics, and a CI recipe.

dlq

Inspect, replay, and discard the dead-letter-queue envelopes a pipeline’s dlq: sink wrote. A DLQ location is a local .jsonl file, a directory of *.jsonl files, or a glob.

faucet dlq inspect ./dlq/breaches.jsonl                          # breakdown + sample
faucet dlq replay pipeline.yaml --from ./dlq/breaches.jsonl --dry-run
faucet dlq replay pipeline.yaml --from ./dlq/breaches.jsonl      # re-feed through the pipeline
faucet dlq discard ./dlq/breaches.jsonl --reason contract --before 7d

faucet dlq inspect <location> — group envelopes by reason and error kind with a sample.

faucet dlq replay <config> --from <location> — re-feed the quarantined payloads through the config’s transforms → quality → contract → sink. Rows that fail again go to a fresh DLQ, never back to the source.

faucet dlq discard <location> — remove processed envelopes.

See the Dead-letter queues cookbook page for the envelope shape and the inspect → fix → replay → discard workflow.

contract

faucet contract pipeline.yaml                       # validate + human summary
faucet contract pipeline.yaml --export contract     # canonical contract JSON
faucet contract pipeline.yaml --export json-schema  # standalone JSON Schema
faucet contract pipeline.yaml --export openlineage  # OpenLineage schema facet

Validates the config’s pipeline.contract: block (a malformed contract exits non-zero with the compile error) and prints a summary of the promised fields, constraints, and breach policy — or, with --export, a machine-readable artifact for downstream consumers. Offline-safe: secrets are never fetched. Requires the contract Cargo feature (in the default build). See the Data contracts cookbook page.

masking

faucet masking pipeline.yaml     # validate + per-destination rule breakdown
faucet masking                   # auto-discover faucet.yaml in cwd

Validates the config’s pipeline.masking: block (a malformed policy exits non-zero with the compile error) and prints, per destination sink, which rules apply — the fast way to confirm applies_to scoping. Offline-safe: secrets are never fetched. Requires the masking Cargo feature (in the default build). See the masking cookbook page.

catalog

(requires the catalog build feature — included in full)

faucet catalog datasets --config pipeline.yaml                 # list catalogued datasets
faucet catalog datasets --config pipeline.yaml --kind csv --q users --json
faucet catalog show 3f2a9c1e0b7d4a55 --config pipeline.yaml    # detail (id prefix ok)
faucet catalog lineage --config pipeline.yaml --root 3f2a9c1e0b7d4a55 --depth 3

Browses the Data Movement Catalog named by the config’s catalog: block: the dataset list (newest activity first, --kind / --q filters), one dataset’s detail (schema timeline with diffs, recent volume, upstream/downstream edges), and the lineage graph. All subcommands accept --json; --config auto-discovers faucet.yaml in cwd when omitted. Read-only — it never mutates the store.

notify

(requires the notify build feature)

faucet notify test pipeline.yaml --event run_failure
faucet notify test --event circuit_open        # auto-discover faucet.yaml

Fires one synthetic event through the config’s notifications: rules using the real delivery path (no pipeline runs) — the fast way to confirm a Slack / PagerDuty / webhook channel is wired correctly. --event accepts any event kind (run_failure, run_success, sla_breach, circuit_open, contract_abort, dlq_threshold, scheduler_stuck). See the Notifications cookbook page.

replicate

faucet replicate pipeline.yaml                 # bulk snapshot, then stream CDC; Ctrl-C to stop
faucet replicate                               # auto-discover faucet.yaml in cwd
faucet replicate pipeline.yaml --env-file prod.env
faucet replicate pipeline.yaml --no-env-file
faucet replicate app.yaml --profile prod       # apply a named profile overlay

Bulk-snapshots a database table and then hands off to change-data-capture from a position captured before the snapshot, producing a true mirror (no gap, no duplicate rows) when paired with write_mode: upsert. The config must contain a top-level replication: block (see config reference); faucet run ignores that block, exactly as it ignores schedule:.

It runs two phases in order:

1. Bulk snapshot — the replication.snapshot.source (a non-CDC query reader) back-fills the destination through the same sink and pipeline-level transforms.
2. CDC handoff — the pipeline.source CDC connector streams every change committed after the captured position over the snapshot baseline.

When replication.continuous is true (the default) the CDC phase is a long-running foreground process — stop it with Ctrl-C or SIGTERM; the in-flight page flushes at the next page boundary before the process exits. With continuous: false it drains CDC once and exits. A durable state backend (file / redis / postgres, not memory) is required so an interrupted run resumes correctly.

Flags:

See the replication cookbook for the correctness model, the resume behaviour, and the per-database retention caveats.

backfill

faucet backfill pipeline.yaml --from 2026-06-01 --to 2026-07-01 --window 1d
faucet backfill pipeline.yaml --from 2026-06-01 --to 2026-07-01 --window 1d --dry-run
faucet backfill pipeline.yaml --from 2026-06-01 --to 2026-07-01 --window 1d --resume
faucet backfill pipeline.yaml --from-bookmark '42' --to-bookmark '99' --bookmark-field seq

Replays a bounded historical window: chunks [from, to) into contiguous half-open window units, runs each through the normal pipeline path with its ${backfill.*} tokens substituted and the ${now.*} clock set to the window start, and records durable, resumable progress in the config’s state: store. Unit state keys are namespaced ({name}::backfill::{unit}) so the forward-sync bookmark is never touched; delivery is forced to at-least-once (pair with write_mode: upsert). Exits non-zero with the failed-unit count.

See the backfill cookbook and the backfill: config block.

schedule

faucet schedule pipeline.yaml                  # run on cron schedule, foreground; Ctrl-C to stop
faucet schedule pipeline.yaml --once           # run exactly once now, then exit
faucet schedule pipeline.yaml --env-file prod.env
faucet schedule pipeline.yaml --no-env-file
faucet schedule app.yaml --profile prod        # schedule with a named profile overlay applied

Runs a pipeline on a recurring cron schedule in a long-running foreground process. The config must contain a top-level schedule: block (without one, faucet errors and suggests faucet run). Requires the schedule Cargo feature (included in full).

• Stop with Ctrl-C or SIGTERM; the in-flight run drains for up to shutdown_grace_secs (default 30) before the process exits.
• --once ignores cron timing and runs the pipeline exactly once immediately — handy for testing a scheduled config or for one-shot container invocations.
• Missed ticks are skipped, not backfilled. A run that starts late emits faucet_schedule_run_lateness_seconds for monitoring.

Flags:

See the scheduling cookbook for worked examples, the overlap-policy decision tree, the resilience/supervisor model, and the full metric set to scrape.

serve

FAUCET_SERVE_AUTH_TOKEN=s3cret faucet serve --listen 0.0.0.0:8080
faucet serve --no-auth                             # explicit opt-in; required if no token
faucet serve --history sqlite:/var/lib/faucet/runs.db --default-config defaults.yaml

Runs a long-running HTTP control plane that accepts pipeline configs over REST, executes them under bounded concurrency (reusing the same executor as faucet run), and exposes status / cancel / list / SSE-logs endpoints plus /healthz, /readyz, and /metrics. Requires the serve Cargo feature (included in full).

Unlike the other commands, serve takes no config file — configs arrive per request. Auth is mandatory: pass --auth-token/FAUCET_SERVE_AUTH_TOKEN, or --no-auth to explicitly disable it (absent both, startup fails).

Selected flags (faucet serve --help for the full list):

Optional embedded web console (serve-ui)

When built with the serve-ui Cargo feature, faucet serve also serves a browser-based web console at / (and static assets at /assets/*):

cargo install faucet-cli --features serve-ui
FAUCET_SERVE_AUTH_TOKEN=s3cret faucet serve --listen 127.0.0.1:8080
# Open http://127.0.0.1:8080/ in a browser.

The static shell is public; all /v1 data is bearer-gated as usual. The browser is prompted for the token on first load; it is stored in localStorage and sent on every /v1 call. Pass --no-ui to disable the console at runtime without rebuilding.

serve-ui implies serve and is included in the full aggregate. It ships three additional bearer-gated endpoints:

These endpoints require serve and are available regardless of --no-ui. See the web console guide for the full walkthrough and the HTTP API reference for the complete endpoint/schema reference.

⚠️ serve executes arbitrary client-supplied configs with the server’s identity (secrets, files, network egress). Run single-tenant, authenticated, behind egress controls. See the serve cookbook for the security model and the HTTP API reference for endpoints.

Environment-only mode

faucet run --from-env assembles a pipeline from a FAUCET_* snapshot (FAUCET_SOURCE_*, FAUCET_SINK_*, FAUCET_STATE_*, FAUCET_TRANSFORM_<N>_*), which is handy for containerized deployments where everything comes from the environment. Nested/tagged-enum fields use a *_JSON suffix.

The complete config grammar (matrix, templates, vars, execution) lives in cli/README.md.