Skip to main content
A continuous source keeps its session open across many commits. The source reports when a portion of the input is complete. The sink reports when its finite write is durable. The runner uses those two promises to decide where a later attempt may safely resume. Start with the streaming lifecycle for the end-to-end picture. This page focuses on the connector’s part in that lifecycle.

Declare what the pair can support

Sources describe whether they emit rows, changes, or messages, which operations they produce, and what ordering and replay guarantees they offer. Continuous planning currently requires replayable, at-least-once input. A best-effort feed cannot promise to recover the messages lost during a worker interruption. Sinks declare one set of write policies for both execution modes. Continuous runs use finite writes through those same policies. The planner checks operations, ordering, durability, and required primary keys before writing. Replace is unavailable for continuous execution because it needs a finite snapshot and a completion boundary. The current planner accepts an order-dependent sink policy only when the source promises strict global order. Partition or transaction ordering alone does not satisfy that requirement. Declare the guarantee the connector can actually maintain through extraction and writing.

Keep the source session alive

Implement StreamSource to open a reusable StreamSession. Opening receives the selected resources, certified resume positions, resource membership, and the admitted worker identity. Use those positions to resume the existing feed; do not silently replace an unavailable position with the latest available one. The runner alternates reads and acknowledgements on the session. A read writes typed rows through StreamRecordSink and returns the source positions covered by that read. Those positions are candidates until the runner has finished the pipeline and committed the epoch. Backpressure can block a row flush for much longer than a network read. Keep protocol heartbeats and any required delivery renewal alive while waiting. Honor cancellation, and join an outstanding read or acknowledgement before closing the session. Closing concurrently with a read is not part of the contract.

Show that the source boundary is complete

Consider a source transaction that updates an order and inserts its line items. Reaching a row limit after the order update is not a safe place to save progress if resuming there would skip the line items. Finish the transaction and report its end before returning coverage. Sources express these boundaries with ordered controls alongside the rows. Transaction begin and end markers describe transaction boundaries; a progress boundary can describe a safe position outside a transaction. Sending a control flushes the affected builders and waits for preceding sink writes and integrity checks. It does not commit the epoch or acknowledge the provider. Before an epoch can be sealed, every covered domain needs an accepted safe boundary after its last row, and no transaction may remain open. Returning a position from Read without the corresponding pipeline evidence is not enough. The source is also responsible for proving it did not skip relevant input on the way to that position. The pipeline cannot discover a missing broker message from the rows it received. Treat record, byte, idle, and age limits as targets for finding that boundary. Cancellation is an interruption, never evidence that a transaction completed. The current runner accepts domain-position coverage; the inbox-claim types in the contracts do not provide an implemented inbox execution path.

Give bookmarks a stable meaning

An ordering domain identifies the part of the source whose positions can be compared, together with its incarnation. A partition offset is useful only if it still refers to the same partition history. Detecting a recreated source prevents an old offset from being mistaken for progress in a new feed. Implement a position codec that validates, compares, and canonicalizes those bookmarks. Codec lookup must work without connector configuration or network access. Register the codecs for runtime persistence as well as exposing them through the source. The datastore needs the same interpretation when it checks for regression or compares a retried certificate. Where the source has a stable event identity, preserve it across retries. It combines the domain, position, and an ordinal for events sharing a position. Do not build that identity from the run or worker attempt: doing so would make a replay look like a new event to a deduplicating sink.

Acknowledge only certified work

Acknowledge receives coverage after destination commit and durable epoch certification. A source must not acknowledge speculative progress during a read. Check current ownership through the supplied authority callback before provider acknowledgements, including redeliveries that you suppress because they are already certified. This check matters when an old worker wakes up after losing its lease. Knowing the correct bookmark does not give it permission to change the provider’s consumer state.

Implement finite sink writes

Bounded and continuous runs use the same sink contract. Open acquires a session. Begin starts a finite write, Apply writes its batches, and Commit confirms their advertised durability. Abort discards tentative work and Close releases session resources, including after a failed Open. Declare write modes once in SinkCapabilities.WritePolicies. Continuous planning checks source operations, keys, ordering, and destination durability. Replacement requires a finite snapshot and is not admitted for continuous runs. Set ReusableSession only after implementing and testing repeated Begin and Commit calls on one open session. Otherwise the runner creates a fresh sink for each boundary, closes it after commit, and restores known schemas on the next instance. This repeats client and schema setup. It does not turn a single-write connector into a reusable connector. The default boundary targets are 1,000 records or one second for reusable sinks and 100,000 records or 30 seconds for fresh instances. An explicit RunOptions.EpochBoundary overrides those defaults. Sources may return a safe boundary earlier or finish an in-progress transaction before returning. RunSpec.Run and StartedAt retain the logical run’s identity and start time. The runner supplies SinkWriteID and SinkWriteStartedAt for each destination session. Use RunSpec.SinkWriteIdentity() for staging and artifact names. It falls back to the logical run ID for bounded runs. A write ID is not a stable event deduplication key. Object sinks retain the logical run ID in templates and manifests, while continuous date partitions use the finite write’s time. The runner owns source epochs, leases, and positions. Sinks receive ordinary write policies without source ownership. The runner verifies every Apply receipt, waits for Commit, persists the certificate, and only then acknowledges the source. Single-write sessions also close before certification. A policy with DurabilityAfterApply confirms each successful batch independently. A policy with DurabilityAfterCommit stages its writes until commit. Continuous planning prefers a compatible commit-durable alternative when one is available. Saved policies must still match a supported alternative exactly. PostgreSQL offers transaction-durable append, upsert, and merge policies for continuous writes. Its bounded policies remain batch-durable. Kafka and NATS acknowledge individual publications during Apply. Both behaviors fit the same sink lifecycle. Abort must preserve previously durable writes. A replay can still duplicate append data if destination commit succeeded before the runner saved its certificate.

Add live message schema support

Implement filament.MessageSchemaSource.OpenMessageStream to opt in. The runner uses this method when the sink implements SchemaPolicySink and enables a policy. There is no separate source capability flag. Ordinary OpenStream retains the connector’s existing behavior. Create one streamkit.MessageBatches with NewMessageBatches(opts, baseSchema) when opening the session. The base schema contains transport columns only. The helper validates saved layouts and owns independent copies. Share this instance across physical readers of the same logical resources and call it only from the serial read owner. Buffer one complete epoch within its record, byte, and time limits. Keep its messages alive and unacknowledged. Before writing transport fields or rows, call the helper with all payloads from that batch.
The helper restores the decoder, probes every payload, applies the policy, and asks the pipeline for the current writer. Use columns.Prepare(payload) to obtain the payload appender, write the transport fields, and finish the row with the existing event metadata projector. Do not keep using an older writer after negotiation. A failed epoch ends the session and discards tentative layouts. The runner persists accepted layouts with certified progress. JSON inference remains in streamkit. It is separate from pipeline admission through MessageSchemaRecordSink. The saved message layout currently supports JSON object fields and raw payloads. Typed database and schema registry sources need an expanded layout contract before using this path. For a sink, implement SchemaPolicySink, which includes Schematized, and return the configured policy from SchemaPolicy. Normalize the requested model to canonical physical types, create the table if needed, and acquire the destination’s lock or transaction. Then call schema.Reconcile with an adapter implementing schema.Target. Inspect returns the complete physical model. AddNullable receives all approved fields together so the adapter can issue one ALTER TABLE. The helper compares columns by name, validates the entire diff, emits detection events, applies the additions, and verifies the result. It never creates or truncates tables and never commits a transaction. The adapter must restore its lock before returning if DDL releases it. A transactional destination can keep DDL with its writes. A destination that commits DDL separately must restore any released lock before verification. A failed run can leave additions behind, so retries inspect the destination without a separate migration state machine. Keep native type mapping and unsupported-feature checks in each connector. Implementing reconciliation alone does not establish continuous sink support. The MySQL bounded sink demonstrates this adapter with independently committed DDL. It applies approved additions together and reacquires its table lock before verification. Continuous runs can use compatible write policies through fresh sink instances. Independently committed DDL is not rolled back with a failed write.

Schema diagnostics

Call schema.Observe with the operation context and the complete diff before applying changes. The reconciler does this automatically for destination diffs. The message helper does this automatically for structural changes to an accepted layout. The runner publishes typed schema.detected events with source or sink origin, policy, field names, and change categories. An initial inferred layout is not drift. Empty diffs produce no event. The runner publishes schema.blocked when a typed block ends an attempt or bounded resource operation. Payload validation failures can block without a structural detection event. Neither event claims that DDL committed. Publication is best effort and may repeat on replay. Events contain no row values, and checkpoint persistence remains the authority for progress. Exercise late fields, omitted fields after restart, strict additions, invalid types later in a batch, failed negotiation, and shared logical resources. The shared helper tests cover inference and admission. Each connector must also test its real transport, acknowledgements, and destination rollback. TestMessageSchemaConnectorIndependentLifecycle in runner/message_schema_integration_test.go exercises the complete shared path with in-memory adapters and serialized certificates. It covers both policies, late fields, restart restoration, event publication, and blocked batches without importing a production connector or opening a database. Run it with go test ./runner -run TestMessageSchemaConnectorIndependentLifecycle.

Make shutdown an honest promise

Close releases the session and joins pending local work. Return an error when remote effects may still be outstanding or their shutdown cannot be established. The runtime records an unproven termination and blocks automatic takeover in that case. An Apply or Commit error does not by itself prove that cleanup failed. Local validation errors can end cleanly. PostgreSQL can also establish clean termination after a confirmed rollback. Lost publication acknowledgements or uncertain commit outcomes need connector-specific handling. The runner relies on the sink’s cleanup result rather than turning every write error into an unproven termination. A clean close does not establish owner fencing or exactly-once delivery.

Test interruptions at the boundaries

Successful delivery is only part of streaming support. Exercise a slow sink, cancellation during a read, a partial source transaction, and loss of ownership before acknowledgement. Verify that a source cannot certify a position whose rows have not finished writing. For a sink, test failed writes, uncertain commit responses, repeated lifecycle calls, and shutdown with outstanding work. An interruption after destination commit but before certificate persistence is especially useful: recovery may replay that epoch, and the result must match the connector’s documented duplicate-handling behavior. The public contracts live in stream.go and epoch.go. The coordinator is in runner/continuous.go, and pipeline completion is enforced in pipeline/epoch.go. The Kafka and NATS sources and PostgreSQL and NATS sinks provide concrete implementations to read alongside those contracts.