Optional interfaces

The base source and sink interfaces are enough for a simple full copy. Other workflows are assembled from small interfaces. Treat each one as a promise: advertising a capability without implementing the contracts it needs usually fails planning or the run; it does not always degrade to a simpler mode.

​

Start from the developer outcome

For example, implementing only CursorColumnProvider makes cursor metadata available, but it does not make extraction incremental. The connector must also plan incremental checkpoints, read from them, and advertise the corresponding source policy.

​

Discovery and schemas

Discoverable.Discover supplies the catalog used before a pipeline exists: resource names, selectors, primary keys, optional schemas, and row estimates. DiscoverOpts.Refresh asks the connector to bypass any cached catalog. SchemaProvider.Schema is the runtime schema contract. It becomes mandatory when the selected sink implements Schematized; without it, the run fails before data moves. Schema fields carry a portable logical type and may also preserve a backend-native type. ResourcePlanner.PlanResources is useful when user-facing selectors do not map one-to-one to emitted resource names. Its output is what Filament uses for schema lookup, policy binding, and checkpoint setup.

​

Snapshot resume and incremental reads

Both workflows eventually call Resumable.ExtractFrom, but their plans come from different places:

• ResumePlanner creates a stable per-resource or per-part plan for continuing the same logical run. A nil checkpoint marks a resource that must be reread.
• IncrementalPlanner starts a later run from a durable cross-run watermark and the pipeline’s cursor configuration.
• CursorColumnProvider describes eligible and recommended cursor fields. It is also used to bind the version field for incremental upserts.

During extraction, each row’s RowMeta.Key identifies checkpoint progress and the writer’s part identifies which planned unit advanced. Implementations should read only from the supplied plan; silently inventing new part layouts makes recovery ambiguous.

​

CDC

ReplicationAware.Replication reports whether a particular connection config is standard or CDC. Without this interface, Filament treats the source as standard. A CDC route then uses ChangeSource.ExtractChanges, supplying selected resources, prior checkpoints, limits, and an observation callback. ChangeAcknowledger is narrower: after sink commit and durable checkpoint observation, the runner can tell the source which upstream positions are safe to release. A source must never acknowledge beyond those positions. CDC support therefore requires agreement between connection mode, advertised source policies, emitted ordered operations, and the extraction interface.

​

Validation and rate limits

Every source already implements the pure Validate method on Source. ConfigValidatable provides the same pure validation hook for types that do not otherwise require it, including sinks. It must not make network calls. LiveValidatable.TestConnection performs an explicit network probe for either a source or sink. RateLimited.Limits lets a source publish its own requests per second and burst ceiling; it describes source behavior rather than replacing run-level rate settings.

​

Sink extensions

The last two interfaces should not be confused with runtime behavior. Today the engine always writes through Sink.Apply and finishes through Commit or Abort. A sink that advertises upsert implements that policy inside Apply. A sink that needs staging manages it inside its own lifecycle.

​

Keep declaration and behavior together

ConnectorSpec.SourcePolicies and SinkSpec.Capabilities are advertisements; the interfaces are executable behavior. Review both whenever you add a mode. Planning checks the advertised policy, write capability, primary keys, and cursor metadata, while the runner later asserts the interfaces required to carry it out. Keeping those two views aligned gives connector users an early, understandable error instead of a failure after extraction begins.