Skip to main content
Filament uses the same connector data path whether a run starts in-process, from the standalone binary, or in a Kubernetes worker. Deployment changes where the work runs, not how connectors exchange data. The lifecycle depends on execution mode. Bounded runs finish with one final commit. Continuous runs keep their sessions open and commit successive epochs, each with its own durable progress record. Both use the Arrow handoff and integrity checks described below.

​
Follow one batch

Suppose a source is reading the users table. It opens a typed Arrow builder for a specific (resource, part), writes rows into it, and flushes when the builder reaches a configured limit. The engine then moves the resulting batch through a bounded channel to a writer, which calls the sink:
A part is a unit of source work, such as a snapshot range or stream partition. The source assigns each batch to a part so its progress can be tracked independently. The engine checks the in-memory batch before and after the sink call. A sink that advertises encoded integrity also returns evidence for the exact bytes it serialized. See Integrity and checkpoints for the boundary each checksum protects.

​
Backpressure and parallelism

The handoff to writers is deliberately bounded. If sink.Apply cannot keep up, the source eventually blocks while flushing a builder. That pressure propagates to extraction and caps queued memory. snapshot_parallelism sets the writer-pool size and is also available to the source when it plans snapshot work. It does not automatically split or hash rows. The source still decides how many parts to create. Filament forces the writer count to one when a route contains CDC, incremental, or other checkpointed work so progress and ordering stay unambiguous.

​
Bounded commits, aborts, and checkpoints

The runner opens the sink before data moves and ends with one of two outcomes:
  • On success, it calls Commit. CDC positions and other progress that depends on a successful commit are saved only after this call succeeds.
  • On a non-resumable failure or cancellation, it calls Abort on a bounded cleanup context.
A resumable extraction failure preserves already durable progress instead of aborting it. Whether a route qualifies depends on every resource in that route, not merely on one incremental table. The recovery rules are described in Runs and recovery.

​
Continuous commits

The continuous coordinator starts an epoch and asks the source to read through a safe boundary. It waits for the pipeline to finish those writes, commits the destination, and saves an epoch certificate before acknowledging the source. It then begins the next epoch using the same open sessions. The source establishes which input is covered; the pipeline establishes that the corresponding writes and integrity checks finished; the sink establishes durability. The coordinator checks that those accounts agree before saving progress. See Streaming for a worked example and the consequences of an interruption between those steps.

​
Facts and stored state

The runner publishes events such as run.started, batch.written, and resource.completed. For bounded runs, the tracker uses those events to update stored progress, totals, and checkpoints. Continuous progress is saved directly through the streaming runtime’s epoch commits. Its activity events report that certified progress; replaying events is not what makes it durable. The control plane writes request, pause, resume, and cancellation state directly. Worker events record what actually happened during execution. Keeping those steps separate lets the API accept a control request while a worker is still responding to it.

​
Where the pieces live

The root filament package defines connector and run contracts. runner coordinates bounded runs and continuous epochs, while pipeline owns the bounded Arrow handoff and writer pool. Control-plane modules handle scheduling, dispatch, tracking, and stale worker detection. The stream supervisor admits continuous worker attempts and reconciles them with the requested execution state. Connectors register independently and implement only the capabilities they advertise.