Source discovery (auto-generate configs)

faucet discover connects to a config’s source, enumerates the datasets living behind it — tables in a database schema, MongoDB collections, Elasticsearch indices, object-store prefixes — and emits a ready-to-run config with one matrix row per dataset. “Replicate this database” becomes one command instead of dozens of hand-written rows.

Quick start

Point a minimal connection config at the system:

# conn.yaml
version: 1
name: warehouse
pipeline:
  source:
    type: postgres
    config:
      connection_url: ${env:DATABASE_URL}
      query: SELECT 1        # placeholder — discovery ignores it
  sink:
    type: jsonl
    config: { path: ./out.jsonl }

faucet discover conn.yaml -o pipeline.yaml
faucet validate pipeline.yaml     # the generated config always validates
faucet run pipeline.yaml

The generated document is your input config with the matrix: block replaced — connection settings, sink, state, auth catalog and everything else pass through untouched, and secrets are echoed as their raw references (${env:…}, ${vault:…}), never as resolved values:

# …your conn.yaml content…

# Generated by `faucet discover` — one row per discovered dataset (2).
matrix:
  # public.orders (table, ~1204 rows)
  #   columns: id integer, note string?, total number
  - id: public_orders
    source:
      config:
        query: SELECT * FROM "public"."orders"
  # sales.leads (table, ~87 rows)
  #   columns: id integer, active boolean?
  - id: sales_leads
    source:
      config:
        query: SELECT * FROM "sales"."leads"

Each row deep-merges a per-dataset config patch over the connection config; introspected column schemas and row estimates appear as comments (? marks a nullable column).

Filters and output

faucet discover conn.yaml --include 'public.*' --exclude '*.tmp_*'  # *-wildcards on dataset names
faucet discover conn.yaml --source warehouse                        # a named pipeline.sources template
faucet discover conn.yaml --json                                    # machine-readable descriptor list
faucet discover conn.yaml -o pipeline.yaml --force                  # overwrite an existing output file

--json emits { "source": "<kind>", "datasets": [ { name, kind, schema?, estimated_rows?, config_patch } ] } for scripting.

Supported sources

Discovery is read-only and cheap — catalog metadata queries and a single listing, never a data scan.

Source	Lists	Schema	Row estimate	Row patch
`postgres`	base tables (all non-system schemas)	`information_schema.columns`	`pg_class.reltuples`	`query`
`mysql`	base tables (current database)	`information_schema.columns`	`table_rows`	`query`
`mssql`	base tables	`INFORMATION_SCHEMA.COLUMNS`	`sys.partitions`	`query`
`sqlite`	tables (`sqlite_master`)	`pragma_table_info`	—	`query`
`mongodb`	collections (non-`system.*`)	inferred from a 10-doc sample	`estimated_document_count`	`collection`
`elasticsearch`	indices (non-`.`-system)	`_mapping` field types	`_cat/indices` docs.count	`index`
`bigquery`	`dataset.table` (physical tables; capped at 500 enumerated / 100 schema fetches, warned)	`tables.get`	`numRows`	`query`
`snowflake`	`schema.table` (base tables)	`information_schema.columns`	`row_count`	`query`
`spanner`	base tables (default schema)	`INFORMATION_SCHEMA.COLUMNS`	—	`query`
`s3`	common prefixes under the configured prefix (one delimiter listing; falls back to per-object entries)	—	—	`prefix`
`gcs`	same as s3	—	—	`prefix` (objects: `object_keys`)

Any other source kind fails with a typed error naming the supported set. Library users can call Source::discover() directly — it returns the same DatasetDescriptor list.

faucet-stream

Source discovery (auto-generate configs)

Quick start

Filters and output

Supported sources

See also

Keyboard shortcuts

faucet-stream

Source discovery (auto-generate configs)

Quick start

Filters and output

Supported sources

See also