Continuous execution and CDC
Execution mode answers how long the work runs. A bounded run reads a finite amount of data and finishes. A continuous run stays active until paused, stopped, or interrupted. Bounded execution is the default. CDC answers a different question: what the source reads. It reads database changes, including inserts, updates, and deletes. A bounded CDC run can catch up to a log position and finish. A continuous message source can stay open without reading a database change log at all. Continuous execution requires a source with a reusable stream session and a datastore that supports the streaming runtime. The sink needs compatible write policies with explicit durability. It does not need a streaming interface. Support for bounded CDC or incremental reads alone does not establish source streaming support. The Kafka and NATS source pages describe their current continuous behavior and limitations. Rows written by a continuous run carry the same_filament_ lineage columns as
a bounded run. CDC sources add the operation, the source position in the form
the source uses, such as a WAL LSN for Postgres or a GTID set for MySQL, and the
sequence within the stream. The
replication modes
guide lists them.
Sink sessions and boundary defaults
A reusable sink keeps its connection open across finite writes. Other sinks receive a fresh instance for each write, including client setup and schema preparation. The source session stays open in both cases.
An explicit
RunOptions.EpochBoundary overrides these defaults. These are soft
targets, not a guaranteed write frequency. Source transaction boundaries,
consumer credit, and available input can change the size of an epoch. Larger
boundaries amortize setup costs for sinks that reopen each time.
The logical run ID stays unchanged. Each destination session gets a separate
write identity for its artifacts. S3 and GCS preserve the logical run ID in
manifests and {{.Run}} templates. Their continuous date partitions use the
finite write’s start time, so a long-running stream can move into a new day.
Follow an order through an epoch
Suppose the last saved position in an orders feed is message 100. The next epoch reads messages 101 through 150 and sends their rows to the destination. Filament then needs to establish three things in order:- All rows through message 150 have finished passing through the pipeline, including the batch integrity checks.
- The destination has confirmed the epoch’s writes are durable according to its connector contract.
- Filament has durably recorded the source position and the destination’s receipts together in its datastore.
Batches and epochs have different jobs
A batch controls how rows move through memory and into the sink. An epoch controls how much work is committed and recorded together. One epoch may contain several batches, and an epoch does not imply atomic visibility at every destination. For example, newly planned PostgreSQL continuous writes use a transaction for each epoch. The NATS sink publishes messages that become visible individually. Finishing the epoch confirms those publishes. It does not turn them into a transaction. Epoch boundaries use soft targets for record count, bytes, idle wait, and age. A source may need to finish a transaction before returning a safe boundary, even if that exceeds a target. An idle read with no progress does not create a certificate. A boundary that safely advances the source position can be certified even when it produces no destination rows. The current coordinator processes epochs serially. Rows still pass through the bounded Arrow pipeline, so a slow destination eventually makes the source wait. The source session must keep its connection and protocol alive during that wait.Where a stream resumes
A stream may have more than one bookmark. Kafka partitions, for example, each have their own position. Filament calls each independently ordered part of a source an ordering domain. A position belongs to that domain and to a particular incarnation of the source, so a recreated feed cannot silently reuse progress from an older one. These bookmarks describe source progress, not a universal ordering across all rows. Reading partitions one at a time does not establish a meaningful global order between them. Filament checks the source’s advertised ordering against the destination’s write requirements before starting. If the orders worker fails before its epoch certificate is saved, the durable bookmark remains at 100. A later attempt may read messages 101 through 150 again. This can happen even if the destination committed them: the worker might have stopped between the destination commit and the datastore write. Recovery is therefore at least once. Append destinations can contain duplicates after replay. A destination may use stable event identities to deduplicate, but the strength and duration of that protection belong to the connector. An epoch certificate does not provide universal exactly-once writes. If the certificate was saved but the source acknowledgement failed, the saved position remains valid. A replacement worker starts with that certified progress; any upstream acknowledgement still requires current ownership.Changing which tables a stream carries
A stream’s identity is its source, its sink and its write mode. The tables it carries, their columns and their destination names are its membership, and membership can change without starting the stream over. Run the pipeline again after saving a version that adds, removes or reshapes a table. The live run drains its current epoch and ends, and the new run takes over the same stream and the same bookmark. A table the stream has not carried yet is read in full first, pinned at one consistent point, and its changes before that point are skipped so the baseline is not replayed. Tables already carried keep streaming from where they were. A removed table leaves the publication. A table whose columns or destination changed is read in full again the same way, as is any table named in the run request’sresync_resources.
The run history shows the handover as one run ending and the next starting.
Until the live run has drained, the successor waits in the scheduled state.
Stopping the pipeline while a successor waits cancels the successor.
Each baseline is certified with the epoch that carries it, so a worker that
fails during one reads it again on the next attempt. Append destinations
receive the baseline rows again after a resync, the same way they do after any
replay.
Pause, stop, and worker ownership
Continuous runs support pause, resume, and stop. A pause or stop first changes the desired state. The worker observes that request and tries to finish the current epoch within a bounded drain period. Until it finishes, the observed state can bedraining. Forced cancellation leaves unfinished work uncertified.
Resume enables a paused execution again. A stopped execution requires a new
start. These controls differ from the cancel behavior of
bounded runs.
Each worker holds a renewable lease. Filament checks ownership when recording
progress and before acknowledging the source. A worker that loses ownership
must stop, even if it still has buffered messages.
Lease expiry alone cannot prove that an old worker has stopped affecting the
destination. If shutdown cannot be confirmed, the execution becomes blocked
and automatic replacement is refused. This prevents a new worker from writing
alongside an old worker whose requests may still be in flight. Cleanly ended
attempts can retry with backoff. Repeated failures without progress eventually
pause the execution. A write error alone does not make cleanup unproven. Local
validation failures and confirmed rollbacks can end cleanly. Schema blocks and
other admission errors still require their own corrective action.
An upgrade restarts running streams the same way. Each attempt records the
release that dispatched it. When the control plane comes up on a newer
release, it asks one stale attempt per reconcile pass to yield. That worker
finishes its current epoch, ends with the reason yielded for upgrade, and
exits. The stream stays enabled, so the next pass starts a fresh attempt on
the current image from the last committed position, with no backoff. Streams
move to the new release one at a time rather than all at once.
When inspecting a continuous run, look at its desired and observed execution
states, reason, and last committed time alongside the logical run status.
Committed totals come from saved epochs. A quiet feed can remain running
without producing a new commit, while a blocked worker can leave the logical
run active without making progress.