Skip to main content
The PostgreSQL source supports full snapshots, timestamp-based incremental reads, and logical-replication CDC. Full reads of tables with primary keys can resume from checkpoints. Keyless tables can be read in full, but an interrupted read starts that table again. The connector is beta.

​
Configuration

​
Read strategies

scan_strategy changes how a full snapshot reads PostgreSQL storage. It does not change whether the pipeline uses a full, incremental, or CDC read. Large tables can be divided into as many as 64 parallel ranges. Filament saves the chosen strategy and range boundaries in the checkpoint, so resuming a run does not silently produce a different read plan.

​
keyset (default)

Reads the primary-key index in order and saves the last key. Integer keys use numeric ranges; UUID, text, and timestamp keys use sampled ranges with roughly equal row counts. This strategy resumes at the exact next row.

​
bitmap

Uses the same key ranges but lets PostgreSQL read each range in physical heap order. This can reduce random I/O when row location and primary-key order are poorly correlated. A range is the smallest recovery unit, so an interruption repeats the current range rather than continuing at an exact row.

​
ctid

Reads physical block ranges and shares one exported snapshot across parallel workers. On resume, Filament checks whether operations such as VACUUM FULL, CLUSTER, or TRUNCATE replaced the table storage. If so, it plans the table again. It also re-reads rows or newly appended blocks when PostgreSQL’s transaction metadata shows that doing so is necessary. Choose this strategy only when that extra replay is acceptable.

​
auto

Uses PostgreSQL statistics to estimate how closely physical row order matches the primary key. A correlation of at least 0.8 in either direction selects keyset; lower correlation selects bitmap. If statistics are missing, Filament runs ANALYZE once and falls back to bitmap if the value is still unavailable. The probe itself does not fail the run.

​
Incremental

Incremental mode requires a primary key and a timestamp/timestamptz NOT NULL cursor column. Filament auto-ranks conventional names (updated_at, modified_at, last_modified_at, last_updated_at, updated, modified) and accepts an explicit per-resource override for anything else. The application must advance the cursor on every insert and update. Static IDs and random UUIDs are not valid watermarks. The watermark is compound, (cursor, pk...), so rows sharing a timestamp are never skipped. Each cycle opens a repeatable-read transaction, captures the high watermark at the snapshot, and pages low < key <= high ordered by (cursor, pk...). A default 300-second lookback rewinds the low bound to absorb late commits and clock skew. The resulting duplicates are handled by upsert. The first run captures the high watermark, then performs a keyset backfill in incremental_backfill mode. On completion it is promoted to steady-state incremental, so changes committed during the backfill are replayed rather than lost. Hard deletes are not observable through an update watermark. Use CDC when deletes must be replicated.

​
CDC

CDC streams pgoutput over a persistent logical replication slot generated for each durable pipeline route. The slot name is managed by Filament and is not user configuration. CDC requires wal_level=logical and a primary key on every selected table. By default the connector creates the publication and keeps its table list current with publish = 'insert, update, delete'. Set manage_publication=false when an administrator provisions it. CDC pipeline runs also require a datastore with durable replication-stream admission. The PostgreSQL datastore provides it; the in-memory datastore does not, and compilation fails with a precondition error instead of sharing an unsafe process-local slot. When a selected table has no CDC checkpoint, the slot is created with an exported snapshot and the table is read in full through that snapshot, so the baseline and the change stream share one consistent point. Run row limits do not apply to this bootstrap read. snapshot_mode=none skips that read. A table without a checkpoint streams from the slot’s consistent point (or the current WAL position when the slot already exists), so the sink starts empty and fills in as rows change. With REPLICA IDENTITY FULL updates carry the full row, so a merge sink upserts rows it has never seen; deletes of unseen rows are no-ops. Steady-state runs are bounded catch-up cycles. Each replays from the oldest resource LSN to pg_current_wal_lsn() at cycle start, then returns. Continuous CDC is a re-request loop, so it can run on a schedule. Zero-change runs still persist the final LSN per resource. A primary-key-changing update is emitted as a delete of the old row plus an insert of the new one. Standby status is reported every 10 seconds, and only the previously durable cursor is acknowledged.
CDC has operational sharp edges:
  • When a successor commits, Filament drops retired slots on the same source connection. If the source connection changed or the pipeline was deleted without a successor, clean up the old database explicitly; its generated consumer name is stored in the control database’s replication_streams.consumer_name column.
  • A “slot is active” error means another client still holds the route’s slot.
  • A checkpoint older than the slot’s confirmed position is a hard error. The WAL was consumed by another client, and the stream cannot be replayed.
  • TRUNCATE of a tracked table fails the run.
  • An update carrying an unchanged TOAST column with no old value is an error. Set REPLICA IDENTITY FULL on that table.

​
Keyless tables

A table without a primary key is still readable in full. It falls back to the ctid path with the ctid as the record id. That read is not resumable, so a resume re-reads the whole table. Incremental and CDC both reject keyless tables, since each requires a primary key. See Replication modes for how modes map to write policies, and Integrity & checkpoints for why plans are frozen at plan time.