Skip to main content
Integrity and checkpoints answer different questions:
  • Integrity: did the batch change while moving through Filament and the sink’s serialization boundary?
  • Checkpoints: after an interruption, where is it safe to begin reading again?
Neither turns every destination into an exactly-once system. The guarantees depend on the source cursor, the sink’s write mode, and when the sink considers an applied batch durable.

​
What a batch verification proves

Filament moves rows internally as typed Arrow batches. Immediately before Sink.Apply, the pipeline calculates a CRC32-C checksum over the batch’s Arrow buffers, its row and column counts, and its insert/update/delete operations. The sink recalculates that checksum at its final in-memory boundary and returns it as WriteReceipt.WriteCRC.
A mismatch means the in-memory batch changed between those boundaries. Filament publishes batch.chunk_divergence and fails the run. This check is sensitive to row order, column order, nulls, values, and operations. It is not based only on row counts.

​
The encoded boundary

Serialization is another place data can change. Sinks that advertise encoded integrity also checksum the exact bytes handed to their transport and return WriteReceipt.EncodedCRC. Filament requires that evidence from those sinks and publishes batch.encoded_integrity_verified when it is present. The two checks cover different risks: These checks do not universally read data back from PostgreSQL, S3, Iceberg, or another destination. Destination acknowledgement, transactions, and read-after-write validation remain connector-specific.

​
How checkpoints move

Suppose an incremental users pipeline reads rows ordered by (updated_at, id). Each flushed batch carries the position of its last row. After the sink accepts the batch, Filament includes that position on the batch.written fact. The tracker combines those deltas into the resource’s current checkpoint.
Filament keeps two checkpoint scopes:
  • Run-scoped checkpoints allow the same logical run to resume a checkpointed snapshot.
  • Route checkpoints carry incremental or CDC progress from one successful run to the next. They are keyed to an immutable pipeline version and route.
The runner creates or reloads the initial checkpoint plan before extraction. That freezes source decisions such as shard boundaries so a resumed attempt does not silently read a different plan.

​
When progress becomes durable

When a checkpoint can be saved depends on when the sink’s work is recoverable:
  • If accepted batches remain recoverable after Apply, a fully checkpointed, idempotent run can resume within the same logical run.
  • Incremental and CDC route checkpoints become durable only at a committed completion or committed pause boundary. A failed run does not advance what the next logical run will use.
  • If a sink makes data durable only at Commit, its progress is not saved before that commit succeeds.
As batches are accepted, the tracker updates the run’s candidate checkpoint and periodically saves progress that is already safe. It also saves safe progress when a resource or run finishes. This reduces repeated work without moving a cursor past data the sink could lose.

​
What recovery looks like

For the users example:
  1. The last successful run ended at (10:30, id=42).
  2. The next run plans a bounded incremental read starting there.
  3. It writes several batches, then loses its worker before commit.
  4. The route checkpoint remains at the last committed boundary.
  5. A resume may repeat already accepted rows, so an upsert sink converges by primary key instead of skipping data.
This is deliberately at-least-once recovery. Filament prefers a safe re-read to advancing a cursor beyond uncommitted work.

​
Types of cursors

Connectors use different cursor formats because databases and APIs expose different stable positions. Examples include ordered keys, completed ranges, incremental watermarks with primary-key tie-breakers, and replication log positions. Those encodings are connector details. As an operator, focus on three questions: can the resource resume, can repeated writes safely converge, and does the sink make data durable after each batch or only when the run commits? See Runs and recovery for the resulting failure, pause, and resume behavior.