> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture

> How data, progress, and control move through a Filament run

Filament uses the same connector data path whether a run starts in-process, from the
standalone binary, or in a Kubernetes worker. Deployment changes where the work
runs, not how connectors exchange data.

The lifecycle depends on execution mode. Bounded runs finish with one final
commit. Continuous runs keep their sessions open and commit successive epochs,
each with its own durable progress record. Both use the Arrow handoff and
integrity checks described below.

## Follow one batch

Suppose a source is reading the `users` table. It opens a typed Arrow builder
for a specific `(resource, part)`, writes rows into it, and flushes when the
builder reaches a configured limit. The engine then moves the resulting batch
through a bounded channel to a writer, which calls the sink:

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
source → typed row builder → bounded Arrow batch → writer pool → sink.Apply
```

A **part** is a unit of source work, such as a snapshot range or stream
partition. The source assigns each batch to a part so its progress can be
tracked independently.

The engine checks the in-memory batch before and after the sink call. A sink
that advertises encoded integrity also returns evidence for the exact bytes it
serialized. See [Integrity and checkpoints](/pages/guides/concepts/integrity-and-checkpoints)
for the boundary each checksum protects.

## Backpressure and parallelism

The handoff to writers is deliberately bounded. If `sink.Apply` cannot keep up,
the source eventually blocks while flushing a builder. That pressure propagates
to extraction and caps queued memory.

`snapshot_parallelism` sets the writer-pool size and is also available to the
source when it plans snapshot work. It does not automatically split or hash
rows. The source still decides how many parts to create. Filament forces the
writer count to one when a route contains CDC, incremental, or other
checkpointed work so progress and ordering stay unambiguous.

## Bounded commits, aborts, and checkpoints

The runner opens the sink before data moves and ends with one of two outcomes:

* On success, it calls `Commit`. CDC positions and other progress that depends
  on a successful commit are saved only after this call succeeds.
* On a non-resumable failure or cancellation, it calls `Abort` on a bounded
  cleanup context.

A resumable extraction failure preserves already durable progress instead of
aborting it. Whether a route qualifies depends on every resource in that route,
not merely on one incremental table. The recovery rules are described in
[Runs and recovery](/pages/guides/concepts/runs-and-recovery).

## Continuous commits

The continuous coordinator starts an epoch and asks the source to read through
a safe boundary. It waits for the pipeline to finish those writes, commits the
destination, and saves an epoch certificate before acknowledging the source.
It then begins the next epoch using the same open sessions.

The source establishes which input is covered; the pipeline establishes that
the corresponding writes and integrity checks finished; the sink establishes
durability. The coordinator checks that those accounts agree before saving
progress. See [Streaming](/pages/guides/concepts/streaming) for a worked example
and the consequences of an interruption between those steps.

## Facts and stored state

The runner publishes events such as `run.started`, `batch.written`, and
`resource.completed`. For bounded runs, the tracker uses those events to update
stored progress, totals, and checkpoints. Continuous progress is saved directly
through the streaming runtime's epoch commits. Its activity events report that
certified progress; replaying events is not what makes it durable.

The control plane writes request, pause, resume, and cancellation state directly.
Worker events record what actually happened during execution. Keeping those
steps separate lets the API accept a control request while a worker is still
responding to it.

```mermaid theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
flowchart LR
    API[API or scheduler] --> State[(run state)]
    API -- requested/control fact --> Bus[(event bus)]
    Bus --> Runner[runner.RunOne]
    Runner -- execution facts --> Bus
    Bus --> Tracker[tracker]
    Tracker --> State
```

## Where the pieces live

The root `filament` package defines connector and run contracts. `runner`
coordinates bounded runs and continuous epochs, while `pipeline` owns the
bounded Arrow handoff and writer pool. Control-plane modules handle scheduling,
dispatch, tracking, and stale worker detection. The stream supervisor admits
continuous worker attempts and reconciles them with the requested execution
state. Connectors register independently and implement only the capabilities
they advertise.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.