Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Dashboards & alerts

faucet ships ready-made Grafana dashboards and Prometheus alert rules built on the metrics every pipeline emits automatically — production observability without hand-building panels. The artifacts live in the repo under observability/ and are kept honest by a CI lint that fails whenever they reference a metric name that no longer exists in the code.

What ships

Dashboard (uid)Focus
faucet-pipeline-overviewRun outcomes + duration percentiles, source/sink throughput and errors by connector, transform in/out, bookmark staleness, effectively-once page skips, state-store traffic. faucet_build_info annotates version rollouts.
faucet-reliabilityRetries / give-ups / circuit-breaker state, DLQ traffic, poison rows, quality quarantines, contract violations, schema drift, PII masking activity, SLA violations, backfill progress.
faucet-scheduleScheduled-run outcomes, heartbeat staleness, next-tick countdown, lateness p95, overlaps, consecutive-failure streak.
faucet-serveControl-plane request rate/latency, run queue, terminal statuses, history degradation, idempotency replays, cluster claims/reclaims, trigger health.

Each dashboard carries a Data source picker and a Pipeline template variable, so they import cleanly into any Grafana ≥ 10.

Alert rules (observability/prometheus/alerts.yml):

AlertFires whenSeverity
FaucetPipelineErrorRateSpike>50% of a pipeline’s runs fail over 15 mcritical
FaucetNoBookmarkProgress (+Critical)no durable bookmark progress for 1 h / 6 hwarning / critical
FaucetSlaViolationsany freshness/volume SLA violation in 1 hwarning
FaucetCircuitBreakerOpenthe resilience breaker stays open 5 mcritical
FaucetStuckSchedulerschedule heartbeat silent for 90 scritical
FaucetScheduleRunLatenesstick lateness p95 > 60 s for 15 mwarning
FaucetConsecutiveScheduleFailures≥3 consecutive failed scheduled runscritical
FaucetServeHistoryDegradedserve’s history backend degraded 5 mcritical
FaucetOtelExportFailures / FaucetLineageEventsDroppedtelemetry/lineage export failinginfo

Quick start with the example stack

The examples Docker stack provisions both automatically:

docker compose -f examples/docker-compose.yml up -d prometheus grafana
  • Grafana: http://localhost:3000 (admin / admin) — the four dashboards are pre-loaded in the faucet folder.
  • Prometheus: http://localhost:9095 — scrapes a faucet process on the host and evaluates the alert rules.

Point faucet’s exporter at it by enabling Prometheus exposition in your config (the compose stack scrapes host port 9464, the default):

observability:
  prometheus:
    listen_addr: 0.0.0.0:9464

Importing into your own Grafana / Prometheus

  • Grafana UI: Dashboards → New → Import → upload a JSON from observability/grafana/. Pick your Prometheus data source when prompted.
  • Grafana provisioning: mount observability/grafana/ and add a file dashboard provider (see examples/infra/grafana/provisioning/).
  • Prometheus: copy observability/prometheus/alerts.yml next to your prometheus.yml and list it under rule_files:.

Staying in sync

cli/tests/observability_artifacts.rs extracts every faucet_* name the dashboards and alerts reference (histogram _bucket/_sum/_count suffixes normalized) and asserts each exists in the source tree. Renaming a metric without updating the artifacts fails the required Test job. Panels group only by the low-cardinality labels (pipeline, row, connector) — never add record keys or run ids.