Declare what the pair can support
Sources describe whether they emit rows, changes, or messages, which operations they produce, and what ordering and replay guarantees they offer. Continuous planning currently requires replayable, at-least-once input. A best-effort feed cannot promise to recover the messages lost during a worker interruption. Sinks declare one set of write policies for both execution modes. Continuous runs use finite writes through those same policies. The planner checks operations, ordering, durability, and required primary keys before writing. Replace is unavailable for continuous execution because it needs a finite snapshot and a completion boundary. The current planner accepts an order-dependent sink policy only when the source promises strict global order. Partition or transaction ordering alone does not satisfy that requirement. Declare the guarantee the connector can actually maintain through extraction and writing.Keep the source session alive
ImplementStreamSource to open a reusable StreamSession. Opening receives
the selected resources, certified resume positions, resource membership, and
the admitted worker identity. Use those positions to resume the existing feed;
do not silently replace an unavailable position with the latest available one.
The runner alternates reads and acknowledgements on the session. A read writes
typed rows through StreamRecordSink and returns the source positions covered
by that read. Those positions are candidates until the runner has finished the
pipeline and committed the epoch.
Backpressure can block a row flush for much longer than a network read. Keep
protocol heartbeats and any required delivery renewal alive while waiting.
Honor cancellation, and join an outstanding read or acknowledgement before
closing the session. Closing concurrently with a read is not part of the
contract.
Show that the source boundary is complete
Consider a source transaction that updates an order and inserts its line items. Reaching a row limit after the order update is not a safe place to save progress if resuming there would skip the line items. Finish the transaction and report its end before returning coverage. Sources express these boundaries with ordered controls alongside the rows. Transaction begin and end markers describe transaction boundaries; a progress boundary can describe a safe position outside a transaction. Sending a control flushes the affected builders and waits for preceding sink writes and integrity checks. It does not commit the epoch or acknowledge the provider. Before an epoch can be sealed, every covered domain needs an accepted safe boundary after its last row, and no transaction may remain open. Returning a position fromRead without the corresponding pipeline evidence is not enough.
The source is also responsible for proving it did not skip relevant input on
the way to that position. The pipeline cannot discover a missing broker message
from the rows it received.
Treat record, byte, idle, and age limits as targets for finding that boundary.
Cancellation is an interruption, never evidence that a transaction completed.
The current runner accepts domain-position coverage; the inbox-claim types in
the contracts do not provide an implemented inbox execution path.
Give bookmarks a stable meaning
An ordering domain identifies the part of the source whose positions can be compared, together with its incarnation. A partition offset is useful only if it still refers to the same partition history. Detecting a recreated source prevents an old offset from being mistaken for progress in a new feed. Implement a position codec that validates, compares, and canonicalizes those bookmarks. Codec lookup must work without connector configuration or network access. Register the codecs for runtime persistence as well as exposing them through the source. The datastore needs the same interpretation when it checks for regression or compares a retried certificate. Where the source has a stable event identity, preserve it across retries. It combines the domain, position, and an ordinal for events sharing a position. Do not build that identity from the run or worker attempt: doing so would make a replay look like a new event to a deduplicating sink.Acknowledge only certified work
Acknowledge receives coverage after destination commit and durable epoch
certification. A source must not acknowledge speculative progress during a
read. Check current ownership through the supplied authority callback before
provider acknowledgements, including redeliveries that you suppress because
they are already certified.
This check matters when an old worker wakes up after losing its lease. Knowing
the correct bookmark does not give it permission to change the provider’s
consumer state.
Implement finite sink writes
Bounded and continuous runs use the same sink contract.Open acquires a
session. Begin starts a finite write, Apply writes its batches, and Commit
confirms their advertised durability. Abort discards tentative work and
Close releases session resources, including after a failed Open.
Declare write modes once in SinkCapabilities.WritePolicies. Continuous
planning checks source operations, keys, ordering, and destination durability.
Replacement requires a finite snapshot and is not admitted for continuous runs.
Set ReusableSession only after implementing and testing repeated Begin
and Commit calls on one open session. Otherwise the runner creates a fresh
sink for each boundary, closes it after commit, and restores known schemas on
the next instance. This repeats client and schema setup. It does not turn a
single-write connector into a reusable connector.
The default boundary targets are 1,000 records or one second for reusable
sinks and 100,000 records or 30 seconds for fresh instances. An explicit
RunOptions.EpochBoundary overrides those defaults. Sources may return a safe
boundary earlier or finish an in-progress transaction before returning.
RunSpec.Run and StartedAt retain the logical run’s identity and start time.
The runner supplies SinkWriteID and SinkWriteStartedAt for each destination
session. Use RunSpec.SinkWriteIdentity() for staging and artifact names. It
falls back to the logical run ID for bounded runs. A write ID is not a stable
event deduplication key. Object sinks retain the logical run ID in templates
and manifests, while continuous date partitions use the finite write’s time.
The runner owns source epochs, leases, and positions. Sinks receive ordinary
write policies without source ownership. The runner verifies every Apply
receipt, waits for Commit, persists the certificate, and only then acknowledges
the source. Single-write sessions also close before certification.
A policy with DurabilityAfterApply confirms each successful batch independently.
A policy with DurabilityAfterCommit stages its writes until commit. Continuous
planning prefers a compatible commit-durable alternative when one is available.
Saved policies must still match a supported alternative exactly.
PostgreSQL offers transaction-durable append, upsert, and merge policies for
continuous writes. Its bounded policies remain batch-durable. Kafka and NATS
acknowledge individual publications during Apply. Both behaviors fit the
same sink lifecycle. Abort must preserve previously durable writes. A replay
can still duplicate append data if destination commit succeeded before the
runner saved its certificate.
Add live message schema support
Implementfilament.MessageSchemaSource.OpenMessageStream to opt in. The runner
uses this method when the sink implements SchemaPolicySink and enables a
policy. There is no separate source capability flag. Ordinary OpenStream
retains the connector’s existing behavior.
Create one streamkit.MessageBatches with NewMessageBatches(opts, baseSchema)
when opening the session. The base schema contains transport columns only.
The helper validates saved layouts and owns independent copies. Share this
instance across physical readers of the same logical resources and call it
only from the serial read owner.
Buffer one complete epoch within its record, byte, and time limits. Keep its
messages alive and unacknowledged. Before writing transport fields or rows,
call the helper with all payloads from that batch.
columns.Prepare(payload) to
obtain the payload appender, write the transport fields, and finish the row
with the existing event metadata projector. Do not keep using an older writer
after negotiation. A failed epoch ends the session and discards tentative
layouts. The runner persists accepted layouts with certified progress.
JSON inference remains in streamkit. It is separate from pipeline admission
through MessageSchemaRecordSink. The saved message layout currently supports
JSON object fields and raw payloads. Typed database and schema registry sources
need an expanded layout contract before using this path.
For a sink, implement SchemaPolicySink, which includes Schematized, and
return the configured policy from SchemaPolicy. Normalize the requested model
to canonical physical types, create the table if needed, and acquire the
destination’s lock or transaction. Then call schema.Reconcile with an adapter
implementing schema.Target.
Inspect returns the complete physical model. AddNullable receives all
approved fields together so the adapter can issue one ALTER TABLE. The helper
compares columns by name, validates the entire diff, emits detection events,
applies the additions, and verifies the result. It never creates or truncates
tables and never commits a transaction. The adapter must restore its lock
before returning if DDL releases it.
A transactional destination can keep DDL with its writes. A destination that
commits DDL separately must restore any released lock before verification.
A failed run can leave additions behind, so retries inspect the destination
without a separate migration state machine. Keep native type mapping and
unsupported-feature checks in each connector. Implementing reconciliation
alone does not establish continuous sink support.
The MySQL bounded sink demonstrates this adapter with independently committed
DDL. It applies approved additions together and reacquires its table lock
before verification. Continuous runs can use compatible write policies through
fresh sink instances. Independently committed DDL is not rolled back with a failed write.
Schema diagnostics
Callschema.Observe with the operation context and the complete diff before
applying changes. The reconciler does this automatically for destination diffs.
The message helper does this automatically for structural
changes to an accepted layout. The runner publishes typed schema.detected
events with source or sink origin, policy, field names, and change categories.
An initial inferred layout is not drift. Empty diffs produce no event.
The runner publishes schema.blocked when a typed block ends an attempt or
bounded resource operation. Payload validation failures can block without a
structural detection event. Neither event claims that DDL committed. Publication
is best effort and may repeat on replay. Events contain no row values, and
checkpoint persistence remains the authority for progress.
Exercise late fields, omitted fields after restart, strict additions, invalid
types later in a batch, failed negotiation, and shared logical resources.
The shared helper tests cover inference and admission. Each connector must
also test its real transport, acknowledgements, and destination rollback.
TestMessageSchemaConnectorIndependentLifecycle in
runner/message_schema_integration_test.go exercises the complete shared path
with in-memory adapters and serialized certificates. It covers both policies,
late fields, restart restoration, event publication, and blocked batches without
importing a production connector or opening a database. Run it with
go test ./runner -run TestMessageSchemaConnectorIndependentLifecycle.
Make shutdown an honest promise
Close releases the session and joins pending local work. Return an error
when remote effects may still be outstanding or their shutdown cannot be
established. The runtime records an unproven termination and blocks automatic
takeover in that case.
An Apply or Commit error does not by itself prove that cleanup failed.
Local validation errors can end cleanly. PostgreSQL can also establish clean
termination after a confirmed rollback. Lost publication acknowledgements or
uncertain commit outcomes need connector-specific handling. The runner relies
on the sink’s cleanup result rather than turning every write error into an
unproven termination. A clean close does not establish owner fencing or
exactly-once delivery.
Test interruptions at the boundaries
Successful delivery is only part of streaming support. Exercise a slow sink, cancellation during a read, a partial source transaction, and loss of ownership before acknowledgement. Verify that a source cannot certify a position whose rows have not finished writing. For a sink, test failed writes, uncertain commit responses, repeated lifecycle calls, and shutdown with outstanding work. An interruption after destination commit but before certificate persistence is especially useful: recovery may replay that epoch, and the result must match the connector’s documented duplicate-handling behavior. The public contracts live instream.go and epoch.go. The coordinator is in
runner/continuous.go, and pipeline completion is enforced in
pipeline/epoch.go. The Kafka and NATS sources and PostgreSQL and NATS sinks
provide concrete implementations to read alongside those contracts.