postgres
The PostgreSQL source supports full snapshots, timestamp-based incremental
reads, and logical-replication CDC. Full reads of tables with primary keys can
resume from checkpoints. Keyless tables can be read in full, but an interrupted
read starts that table again. The connector is beta.
Configuration
Read strategies
scan_strategy changes how a full snapshot reads PostgreSQL storage. It does
not change whether the pipeline uses a full, incremental, or CDC read.
Large tables can be divided into as many as 64 parallel ranges. Filament saves
the chosen strategy and range boundaries in the checkpoint, so resuming a run
does not silently produce a different read plan.
keyset (default)
Reads the primary-key index in order and saves the last key. Integer keys use numeric ranges; UUID, text, and timestamp keys use sampled ranges with roughly equal row counts. This strategy resumes at the exact next row.bitmap
Uses the same key ranges but lets PostgreSQL read each range in physical heap order. This can reduce random I/O when row location and primary-key order are poorly correlated. A range is the smallest recovery unit, so an interruption repeats the current range rather than continuing at an exact row.ctid
Reads physical block ranges and shares one exported snapshot across parallel workers. On resume, Filament checks whether operations such asVACUUM FULL,
CLUSTER, or TRUNCATE replaced the table storage. If so, it plans the table
again. It also re-reads rows or newly appended blocks when PostgreSQL’s
transaction metadata shows that doing so is necessary. Choose this strategy
only when that extra replay is acceptable.
auto
Uses PostgreSQL statistics to estimate how closely physical row order matches the primary key. A correlation of at least0.8 in either direction selects
keyset; lower correlation selects bitmap. If statistics are missing, Filament
runs ANALYZE once and falls back to bitmap if the value is still unavailable.
The probe itself does not fail the run.
Incremental
Incremental mode requires a primary key and atimestamp/timestamptz
NOT NULL cursor column. Filament auto-ranks conventional names
(updated_at, modified_at, last_modified_at, last_updated_at,
updated, modified) and accepts an explicit per-resource override for
anything else. The application must advance the cursor on every insert and
update. Static IDs and random UUIDs are not valid watermarks.
The watermark is compound, (cursor, pk...), so rows sharing a timestamp are
never skipped. Each cycle opens a repeatable-read transaction, captures the
high watermark at the snapshot, and pages low < key <= high ordered by
(cursor, pk...). A default 300-second lookback rewinds the low bound to
absorb late commits and clock skew. The resulting duplicates are handled by
upsert.
The first run captures the high watermark, then performs a keyset backfill in
incremental_backfill mode. On completion it is promoted to steady-state
incremental, so changes committed during the backfill are replayed rather
than lost. Hard deletes are not observable through an update watermark. Use
CDC when deletes must be replicated.
CDC
CDC streamspgoutput over a persistent logical replication slot generated
for each durable pipeline route. The slot name is managed by Filament and is
not user configuration. CDC requires wal_level=logical and a primary key on
every selected table. By
default the connector creates the publication and keeps its table list
current with publish = 'insert, update, delete'. Set
manage_publication=false when an administrator provisions it.
CDC pipeline runs also require a datastore with durable replication-stream
admission. The PostgreSQL datastore provides it; the in-memory datastore does
not, and compilation fails with a precondition error instead of sharing an
unsafe process-local slot.
In bounded execution, when a selected table has no CDC checkpoint, the slot is created with an
exported snapshot and the table is read in full through that snapshot, so the
baseline and the change stream share one consistent point. Run row limits do
not apply to this bootstrap read.
snapshot_mode=none skips that read. A table without a checkpoint streams
from the slot’s consistent point (or the current WAL position when the slot
already exists), so the sink starts empty and fills in as rows change. With
REPLICA IDENTITY FULL updates carry the full row, so a merge sink upserts
rows it has never seen; deletes of unseen rows are no-ops.
Bounded runs are catch-up cycles. Each replays from the oldest resource LSN
to pg_current_wal_lsn() at cycle start, then returns. Schedule these runs
when periodic catch-up is sufficient.
Zero-change runs still persist the final LSN per resource. A
primary-key-changing update is emitted as a delete of the old row plus an
insert of the new one. Standby status is reported every 10 seconds, and only
the previously durable cursor is acknowledged.
Continuous execution
Continuous CDC keeps one logical replication session open and commits finite write boundaries as changes arrive. It does not repeatedly launch bounded runs. To enable it, set the source connection’sreplication to cdc, choose
Continuous execution for the pipeline, and select the tables explicitly. Use merge for current table state
or append for change history when the sink accepts CDC operations.
The control datastore must support continuous runs. The PostgreSQL datastore
provides that capability. Each selected table needs a primary key, and the
source database needs logical replication enabled. No streaming-specific sink
implementation is required.
The default snapshot_mode=initial copies existing rows from a consistent
snapshot, then streams changes retained from that snapshot’s WAL position.
The entire snapshot completes as one write boundary, even if it exceeds the
configured boundary limits. A restart before its first checkpoint repeats the
snapshot and catches up from retained WAL before certifying progress.
Set snapshot_mode=none to start with changes only. After a checkpoint is
certified, restarts resume from that position without repeating the snapshot.
A missing slot or unavailable resume position stops the worker.
Write boundaries end only after complete source transactions. A transaction
larger than the configured row or byte target stays in one boundary. Source
acknowledgments follow destination durability and checkpoint certification.
Pause drains the current boundary. Resume uses the saved position. Stop ends
the execution. Continuous pipelines do not use cron schedules.
Schema changes block the worker when its cached layout no longer matches the
replication relation. Review the schema before restarting. Append writes can
repeat after a failure between destination success and checkpoint persistence.