Skip to main content
A continuous run keeps listening for new data and commits progress along the way. Imagine an orders feed that receives messages throughout the day. There is no final message to wait for, so Filament repeatedly finishes a small portion of the feed, saves where it reached, and begins the next portion. Each of those portions is an epoch. An epoch gives an ongoing stream a recovery boundary without requiring the connection to close or the run to end.

Continuous execution and CDC

Execution mode answers how long the work runs. A bounded run reads a finite amount of data and finishes. A continuous run stays active until paused, stopped, or interrupted. Bounded execution is the default. CDC answers a different question: what the source reads. It reads database changes, including inserts, updates, and deletes. A bounded CDC run can catch up to a log position and finish. A continuous message source can stay open without reading a database change log at all. Continuous execution requires a source with a reusable stream session and a datastore that supports the streaming runtime. The sink needs compatible write policies with explicit durability. It does not need a streaming interface. Support for bounded CDC or incremental reads alone does not establish source streaming support. The Kafka and NATS source pages describe their current continuous behavior and limitations. Rows written by a continuous run carry the same _filament_ lineage columns as a bounded run. CDC sources add the operation, the source position in the form the source uses, such as a WAL LSN for Postgres or a GTID set for MySQL, and the sequence within the stream. The replication modes guide lists them.

Sink sessions and boundary defaults

A reusable sink keeps its connection open across finite writes. Other sinks receive a fresh instance for each write, including client setup and schema preparation. The source session stays open in both cases. An explicit RunOptions.EpochBoundary overrides these defaults. These are soft targets, not a guaranteed write frequency. Source transaction boundaries, consumer credit, and available input can change the size of an epoch. Larger boundaries amortize setup costs for sinks that reopen each time. The logical run ID stays unchanged. Each destination session gets a separate write identity for its artifacts. S3 and GCS preserve the logical run ID in manifests and {{.Run}} templates. Their continuous date partitions use the finite write’s start time, so a long-running stream can move into a new day.

Follow an order through an epoch

Suppose the last saved position in an orders feed is message 100. The next epoch reads messages 101 through 150 and sends their rows to the destination. Filament then needs to establish three things in order:
  1. All rows through message 150 have finished passing through the pipeline, including the batch integrity checks.
  2. The destination has confirmed the epoch’s writes are durable according to its connector contract.
  3. Filament has durably recorded the source position and the destination’s receipts together in its datastore.
Only then may the source acknowledge that progress upstream. For a message broker, that acknowledgement can tell it that these messages no longer need to be delivered to this consumer. The saved record is called an epoch certificate. It ties the source positions, destination receipts, and completed pipeline work to a particular stream and worker attempt. It is the durable evidence used for progress and recovery; a successful batch write alone cannot advance continuous progress.

Batches and epochs have different jobs

A batch controls how rows move through memory and into the sink. An epoch controls how much work is committed and recorded together. One epoch may contain several batches, and an epoch does not imply atomic visibility at every destination. For example, newly planned PostgreSQL continuous writes use a transaction for each epoch. The NATS sink publishes messages that become visible individually. Finishing the epoch confirms those publishes. It does not turn them into a transaction. Epoch boundaries use soft targets for record count, bytes, idle wait, and age. A source may need to finish a transaction before returning a safe boundary, even if that exceeds a target. An idle read with no progress does not create a certificate. A boundary that safely advances the source position can be certified even when it produces no destination rows. The current coordinator processes epochs serially. Rows still pass through the bounded Arrow pipeline, so a slow destination eventually makes the source wait. The source session must keep its connection and protocol alive during that wait.

Where a stream resumes

A stream may have more than one bookmark. Kafka partitions, for example, each have their own position. Filament calls each independently ordered part of a source an ordering domain. A position belongs to that domain and to a particular incarnation of the source, so a recreated feed cannot silently reuse progress from an older one. These bookmarks describe source progress, not a universal ordering across all rows. Reading partitions one at a time does not establish a meaningful global order between them. Filament checks the source’s advertised ordering against the destination’s write requirements before starting. If the orders worker fails before its epoch certificate is saved, the durable bookmark remains at 100. A later attempt may read messages 101 through 150 again. This can happen even if the destination committed them: the worker might have stopped between the destination commit and the datastore write. Recovery is therefore at least once. Append destinations can contain duplicates after replay. A destination may use stable event identities to deduplicate, but the strength and duration of that protection belong to the connector. An epoch certificate does not provide universal exactly-once writes. If the certificate was saved but the source acknowledgement failed, the saved position remains valid. A replacement worker starts with that certified progress; any upstream acknowledgement still requires current ownership.

Changing which tables a stream carries

A stream’s identity is its source, its sink and its write mode. The tables it carries, their columns and their destination names are its membership, and membership can change without starting the stream over. Run the pipeline again after saving a version that adds, removes or reshapes a table. The live run drains its current epoch and ends, and the new run takes over the same stream and the same bookmark. A table the stream has not carried yet is read in full first, pinned at one consistent point, and its changes before that point are skipped so the baseline is not replayed. Tables already carried keep streaming from where they were. A removed table leaves the publication. A table whose columns or destination changed is read in full again the same way, as is any table named in the run request’s resync_resources. The run history shows the handover as one run ending and the next starting. Until the live run has drained, the successor waits in the scheduled state. Stopping the pipeline while a successor waits cancels the successor. Each baseline is certified with the epoch that carries it, so a worker that fails during one reads it again on the next attempt. Append destinations receive the baseline rows again after a resync, the same way they do after any replay.

Pause, stop, and worker ownership

Continuous runs support pause, resume, and stop. A pause or stop first changes the desired state. The worker observes that request and tries to finish the current epoch within a bounded drain period. Until it finishes, the observed state can be draining. Forced cancellation leaves unfinished work uncertified. Resume enables a paused execution again. A stopped execution requires a new start. These controls differ from the cancel behavior of bounded runs. Each worker holds a renewable lease. Filament checks ownership when recording progress and before acknowledging the source. A worker that loses ownership must stop, even if it still has buffered messages. Lease expiry alone cannot prove that an old worker has stopped affecting the destination. If shutdown cannot be confirmed, the execution becomes blocked and automatic replacement is refused. This prevents a new worker from writing alongside an old worker whose requests may still be in flight. Cleanly ended attempts can retry with backoff. Repeated failures without progress eventually pause the execution. A write error alone does not make cleanup unproven. Local validation failures and confirmed rollbacks can end cleanly. Schema blocks and other admission errors still require their own corrective action. An upgrade restarts running streams the same way. Each attempt records the release that dispatched it. When the control plane comes up on a newer release, it asks one stale attempt per reconcile pass to yield. That worker finishes its current epoch, ends with the reason yielded for upgrade, and exits. The stream stays enabled, so the next pass starts a fresh attempt on the current image from the last committed position, with no backoff. Streams move to the new release one at a time rather than all at once. When inspecting a continuous run, look at its desired and observed execution states, reason, and last committed time alongside the logical run status. Committed totals come from saved epochs. A quiet feed can remain running without producing a new commit, while a blocked worker can leave the logical run active without making progress.

Building streaming support

The streaming connector guide explains how sources establish safe boundaries and how sinks participate in the epoch lifecycle. For the checks applied to every batch, see Integrity and checkpoints.