Optional interfaces

The base source and sink interfaces are enough for a simple full copy. Other workflows are assembled from small interfaces. Treat each one as a promise: advertising a capability without implementing the contracts it needs usually fails planning or the run; it does not always degrade to a simpler mode.

​

Start from the developer outcome

For example, implementing only CursorColumnProvider makes cursor metadata available, but it does not make extraction incremental. The connector must also plan incremental checkpoints, read from them, and advertise the corresponding source policy.

​

Discovery and schemas

Discoverable.Discover supplies the catalog used before a pipeline exists: resource names, selectors, primary keys, optional schemas, and row estimates. DiscoverOpts.Refresh asks the connector to bypass any cached catalog. SchemaProvider.Schema is the runtime schema contract. It becomes mandatory when the selected sink implements Schematized; without it, the run fails before data moves. Schema fields carry a portable logical type and may also preserve a backend-native type. ResourcePlanner.PlanResources is useful when user-facing selectors do not map one-to-one to emitted resource names. Its output is what Filament uses for schema lookup, policy binding, and checkpoint setup.

​

Snapshot resume and incremental reads

Both workflows eventually call Resumable.ExtractFrom, but their plans come from different places:

• ResumePlanner creates a stable per-resource or per-part plan for continuing the same logical run. A nil checkpoint marks a resource that must be reread.
• IncrementalPlanner starts a later run from a durable cross-run watermark and the pipeline’s cursor configuration.
• CursorColumnProvider describes eligible and recommended cursor fields. It is also used to bind the version field for incremental upserts.

During extraction, each row’s RowMeta.Key identifies checkpoint progress and the writer’s part identifies which planned unit advanced. Implementations should read only from the supplied plan; silently inventing new part layouts makes recovery ambiguous.

​

CDC

A connection is CDC when the source spec declares a CDC source policy and the connection config sets replication to cdc. Filament derives this with ReplicationFor; a source without a CDC policy is always standard. A CDC route then uses ChangeSource.ExtractChanges, supplying selected resources, prior checkpoints, limits, and an observation callback. ChangeAcknowledger is narrower: after sink commit and durable checkpoint observation, the runner can tell the source which upstream positions are safe to release. A source must never acknowledge beyond those positions. CDC support therefore requires agreement between connection mode, advertised source policies, emitted ordered operations, and the extraction interface. Continuous execution uses a reusable source session and repeated sink epochs. Those contracts are described in Streaming connectors, including how to report complete source boundaries and acknowledge certified progress.

​

Validation and rate limits

Every source already implements the pure Validate method on Source. ConfigValidatable provides the same pure validation hook for types that do not otherwise require it, including sinks. It must not make network calls. LiveValidatable.TestConnection performs an explicit network probe for either a source or sink. RateLimited.Limits lets a source publish its own requests per second and burst ceiling; it describes source behavior rather than replacing run-level rate settings.

​

Sink extensions

The last interface should not be confused with runtime behavior. Today the bounded engine writes through Sink.Apply and finishes through Commit or Abort. Continuous execution uses BeginEpoch, CommitEpoch, and AbortEpoch. A sink that declares an upsert policy implements it inside Apply. A sink that needs staging manages it inside its own lifecycle.

​

Keep declaration and behavior together

ConnectorSpec.SourcePolicies and SinkSpec.Capabilities are advertisements; the interfaces are executable behavior. Review both whenever you add a mode. Planning checks the advertised policy, write capability, primary keys, and cursor metadata, while the runner later asserts the interfaces required to carry it out. Keeping those two views aligned gives connector users an early, understandable error instead of a failure after extraction begins.