> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrity & checkpoints

> What Filament verifies, when progress becomes durable, and where the guarantees end

Integrity and checkpoints answer different questions:

* **Integrity:** did the batch change while moving through Filament and the
  sink's serialization boundary?
* **Checkpoints:** after an interruption, where is it safe to begin reading
  again?

Neither turns every destination into an exactly-once system. The guarantees
depend on the source cursor, the sink's write mode, and when the sink considers
an applied batch durable.

## What a batch verification proves

Filament moves rows internally as typed Arrow batches. Immediately before
`Sink.Apply`, the pipeline calculates a CRC32-C checksum over the batch's Arrow
buffers, its row and column counts, and its insert/update/delete operations.
The sink recalculates that checksum at its final in-memory boundary and returns
it as `WriteReceipt.WriteCRC`.

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
Arrow batch
  ├─ pipeline CRC
  └─ sink CRC ── compare before accepting the batch
```

A mismatch means the in-memory batch changed between those boundaries. Filament
publishes `batch.chunk_divergence` and fails the run.

This check is sensitive to row order, column order, nulls, values, and
operations. It is not based only on row counts.

## The encoded boundary

Serialization is another place data can change. Sinks that advertise encoded
integrity also checksum the exact bytes handed to their transport and return
`WriteReceipt.EncodedCRC`. Filament requires that evidence from those sinks and
publishes `batch.encoded_integrity_verified` when it is present.

The two checks cover different risks:

| Evidence | Boundary covered |
| - | - |
| `WriteCRC` | The borrowed Arrow batch across the call to `Apply` |
| `EncodedCRC` | The sink's final serialized payload before transport |

These checks do **not** universally read data back from PostgreSQL, S3,
Iceberg, or another destination. Destination acknowledgement, transactions,
and read-after-write validation remain connector-specific.

## How checkpoints move

Suppose an incremental `users` pipeline reads rows ordered by
`(updated_at, id)`. Each flushed batch carries the position of its last row.
After the sink accepts the batch, Filament includes that position on the
`batch.written` fact. The tracker combines those deltas into the resource's
current checkpoint.

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
source position → batch.written → tracker → durable checkpoint
```

For bounded runs, Filament keeps two checkpoint scopes:

* **Run-scoped checkpoints** allow the same logical run to resume a
  checkpointed snapshot.
* **Route checkpoints** carry incremental or CDC progress from one successful
  run to the next. They are keyed to an immutable pipeline version and route.

The runner creates or reloads the initial checkpoint plan before extraction.
That freezes source decisions such as shard boundaries so a resumed attempt
does not silently read a different plan.

## When progress becomes durable

The rules below describe bounded runs. Continuous runs use the same batch
integrity checks, but save progress through an epoch certificate after the
destination commits. The certificate connects completed pipeline work to
source positions and destination receipts. Only then can the source
acknowledge upstream progress. See
[Streaming](/pages/guides/concepts/streaming) for that sequence and its replay
boundaries.

When a checkpoint can be saved depends on when the sink's work is recoverable:

* If accepted batches remain recoverable after `Apply`, a fully checkpointed,
  idempotent run can resume within the same logical run.
* Incremental and CDC route checkpoints become durable only at a committed
  completion or committed pause boundary. A failed run does not advance what
  the next logical run will use.
* If a sink makes data durable only at `Commit`, its progress is not saved
  before that commit succeeds.

As batches are accepted, the tracker updates the run's candidate checkpoint and
periodically saves progress that is already safe. It also saves safe progress
when a resource or run finishes. This reduces repeated work without moving a
cursor past data the sink could lose.

## What recovery looks like

For the `users` example:

1. The last successful run ended at `(10:30, id=42)`.
2. The next run plans a bounded incremental read starting there.
3. It writes several batches, then loses its worker before commit.
4. The route checkpoint remains at the last committed boundary.
5. A resume may repeat already accepted rows, so an upsert sink converges by
   primary key instead of skipping data.

This is deliberately at-least-once recovery. Filament prefers a safe re-read to
advancing a cursor beyond uncommitted work.

## Types of cursors

Connectors use different cursor formats because databases and APIs expose
different stable positions. Examples include ordered keys, completed
ranges, incremental watermarks with primary-key tie-breakers, and replication
log positions.

Those encodings are connector details. As an operator, focus on three
questions: can the resource resume, can repeated writes safely converge, and
does the sink make data durable after each batch or only when the run commits?

See [Runs and recovery](/pages/guides/concepts/runs-and-recovery) for the
resulting failure, pause, and resume behavior.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.