Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Try it locally (interactive demo)

The repo ships a single script — scripts/try-local.sh — that builds the faucet CLI, generates a throwaway demo workspace, exercises a broad slice of the toolkit against file-only connectors (no Docker, no cloud, no databases), and then leaves the web console running so you can browse the results visually.

It’s the fastest way to see pipelines, transforms, data-quality, masking, lineage, the Data Movement Catalog, and dead-letter-queue replay working end-to-end on your machine.

Prerequisites

The default build is light and pure-Rust — it needs only:

  • rustup with the toolchain pinned in rust-toolchain.toml (the script resolves it automatically, even if a Homebrew rustc is on your PATH).
  • A C toolchain for a couple of transitive crates — on macOS that’s the Xcode Command Line Tools (xcode-select --install); on Linux, build-essential.

sqlite3 and curl are used by a few steps if present (both ship on macOS and most Linux distros); missing ones are skipped gracefully.

The optional --full build additionally compiles Kafka, gRPC, the cloud connectors, and the DuckDB SQL transform from source, which requires CMake and takes ~15–30 minutes. The light default builds in a few minutes.

Running it

# From the repo root — builds the light feature set, runs the battery,
# then starts the web console and leaves it up (Ctrl+C to stop).
./scripts/try-local.sh

Useful flags:

FlagEffect
(none)Light build → run battery → keep the web console running
--fullBuild every feature (Kafka, gRPC, cloud, DuckDB SQL); needs CMake
--releaseOptimised build (slower to compile, faster to run)
--no-serveRun the battery and exit — no console (for CI / a quick check)
--serve-onlySkip the build + battery; just (re)launch the populated console
--cleanWipe the demo workspace (faucet-local-demo/) first
--no-buildReuse an already-built binary
--port NConsole / serve port (default 8899)

What it exercises

Everything below runs against generated CSV data in faucet-local-demo/:

  • Core: CSV → JSONL, record transforms (set / cast / redact / flatten / filter / explode / value_case), preview, validate, doctor.
  • Governance: data-quality checks + DLQ quarantine, data contracts (quarantine and fail policies), PII masking (redact / hash / partial / tokenize).
  • Round-trips: CSV ↔ SQLite, CSV ↔ Parquet.
  • Runtime: matrix fan-out, depends_on ordering, --from-env, config composition (extends: + profiles:), JSON-format configs, schema-drift evolve (SQLite ADD COLUMN).
  • Observability: SLA monitoring, file-based OpenLineage emission, the Data Movement Catalog.
  • Ops: offline faucet test, dlq inspect / replay, and the serve HTTP control plane.

With --full, the embedded DuckDB SQL transform step is included too.

The web console

When the battery finishes, the script submits a handful of demo runs through the HTTP API and then keeps the server up, so the console arrives already populated — you can open it and immediately browse Runs, Datasets, Lineage, and the per-run dead-letter-queue panel. See Web console for a screenshot tour.

The run-history database (faucet-local-demo/faucet-meta.db) is not wiped between invocations, so run history accumulates over time. Use --clean to reset the whole workspace.

The faucet-local-demo/ workspace is disposable — delete it any time with rm -rf faucet-local-demo. It is git-ignored.