The one-sentence idea

faucet-stream moves data from one place to another.

Picture a kitchen faucet: water comes from a pipe (the source), flows through the tap, and out into the sink. faucet-stream is the tap — you say where the data comes from and where it goes, and it moves the data reliably, without losing or scrambling it.

flowchart LR
    S["Source"] -->|records| P["faucet pipeline"] -->|records| K["Sink"]

Everything else — pages, bookmarks, retries, exactly-once — exists to keep that one sentence true even when things go wrong. We’ll add those ideas one at a time.

Chapter 1 — The two characters: Source and Sink

The whole system is built from two roles:

• A Source knows how to read records from somewhere (a database, an API, a file, a queue).
• A Sink knows how to write them somewhere else.

A connector is just a Source or Sink for one system (faucet-source-postgres, faucet-sink-bigquery, …). They all speak the same two-role language, which is why any source can feed any sink.

Records are just JSON. A database row, an API response, a file line — they all become plain JSON objects flowing through the pipe. At its simplest, a Source is one function (“give me your records”) and a Sink is one function (“here are records, write them”). That’s a working connector; everything else is optional.

Chapter 2 — Moving data once

Connect a Source to a Sink and you have a pipeline: read everything, write everything.

flowchart LR
    A["source.fetch<br/>read all"] --> B["sink.write"] --> C["done — wrote N records"]

For a one-time copy, this is all you need. Two real-world problems push us further: you don’t want to re-copy everything every run (Chapter 3), and your data might be too big for memory (Chapter 4).

Chapter 3 — Only the new stuff (incremental)

To avoid re-reading everything each run, the Source leaves itself a note — a bookmark — saying “I got up to here” (a timestamp, a log position, an offset). Next run it resumes from that note instead of the beginning.

Here’s the single most important rule in the whole project, and it’s just common sense:

The bookmark is saved only after the data is safely written.

If we saved “got to row 1000” first and then crashed before writing those rows, they’d be lost forever. So the order is always write → make sure it’s really saved → then save the bookmark. Crash in between, and the worst case is redoing a little work (safe) — never skipping data (catastrophic). Keep this rule in your pocket; every advanced feature respects it.

Chapter 4 — Bigger than memory (streaming)

Reading a billion rows into memory won’t work. So instead of “all the data,” the Source produces a stream of pages — chunks of, say, 1,000 records at a time — and the pipeline handles one page at a time:

flowchart LR
    P1["page 1"] --> W1["write"] --> P2["page 2"] --> W2["write"] --> P3["page 3<br/>+ bookmark"] --> W3["write"] --> F["flush"] --> CK["save bookmark"]

Only one page is ever in memory, so a thousand rows or a billion, memory stays flat. The bookmark rides along on the pages, and it’s still saved after the page is safely written — Chapter 3’s rule, now per-page.

Chapter 5 — The production toolbox (reach for these when you need them)

You now understand the spine: a source streams pages, the pipeline writes each page and checkpoints safely, so you can resume after a crash. Everything below is optional — a toolbox you pull from the day you hit the problem a tool solves. Find your situation, then follow the tool to its how-to. The family almost every real pipeline reaches for — shaping the data — comes first.

Shaping the data

Guarding the data

Moving it reliably

Getting data in and out at scale

Running & operating it

Seeing what happened

When several of the data-guarding tools are on, each page runs them in a fixed, safe order — mask first (so PII can’t leak), then validate (so bad data never lands), then write, then save the bookmark last:

flowchart LR
    PAGE["page"] --> M["mask"] --> Q["quality"] --> C["contract"] --> D["drift"] --> W["write"] --> FL["flush"] --> CK["save bookmark"]

The golden rule never bends, no matter how many tools you add.

The one rule that ties it all together

A bookmark is saved only after the sink has durably written and flushed the page. Write → flush → checkpoint. Always.

Every failure mode, retry, and exactly-once guarantee is a consequence of that one ordering.

Where to go next

• Run a real pipeline: Your first pipeline.
• The concepts, precisely: Core concepts.
• The full story with diagrams and code: the beginner guide on GitHub.
• Flip this page to 🏛 Architect reference for the condensed deep view.

flowchart LR
    cfg["config"] --> comp["compose"] --> interp["interpolate"] --> sec["secrets"] --> parse["parse"] --> exp["expand"] --> exe["executor"] --> pipe["Pipeline"] --> rs["run_stream"]

flowchart LR
    PAGE["page"] --> M["mask"] --> Q["quality"] --> C["contract"] --> D["drift"] --> WR["write path"] --> FL["flush"] --> CK["checkpoint"]