> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Optional interfaces

> Add discovery, schemas, recovery, and CDC behavior to a connector

The base source and sink interfaces are enough for a simple full copy. Other
workflows are assembled from small interfaces. Treat each one as a promise:
advertising a capability without implementing the contracts it needs usually
fails planning or the run; it does not always degrade to a simpler mode.

## Start from the developer outcome

| You want to support | Source contracts involved |
| - | - |
| Browse resources before building a pipeline | `Discoverable` |
| Create typed destination tables | `SchemaProvider` (when paired with a `Schematized` sink) |
| Resume a checkpointed snapshot | `Resumable` and `ResumePlanner` |
| Read incrementally across runs | `Resumable`, `IncrementalPlanner`, and usually `CursorColumnProvider` |
| Read a database change stream | `ChangeSource` and a CDC source policy in the spec |
| Keep a source session open across continuous commits | `StreamSource`, `StreamSession`, and registered position codecs |
| Release upstream CDC retention after durability | `ChangeAcknowledger` |
| Resolve selectors into concrete resources | `ResourcePlanner` |
| Declare a source-side request ceiling | `RateLimited` |

For example, implementing only `CursorColumnProvider` makes cursor metadata
available, but it does not make extraction incremental. The connector must also
plan incremental checkpoints, read from them, and advertise the corresponding
source policy.

## Discovery and schemas

`Discoverable.Discover` supplies the catalog used before a pipeline exists:
resource names, selectors, primary keys, optional schemas, and row estimates.
`DiscoverOpts.Refresh` asks the connector to bypass any cached catalog.

`SchemaProvider.Schema` is the runtime schema contract. It becomes mandatory
when the selected sink implements `Schematized`; without it, the run fails before
data moves. Schema fields carry a portable logical type and may also preserve a
backend-native type.

`ResourcePlanner.PlanResources` is useful when user-facing selectors do not map
one-to-one to emitted resource names. Its output is what Filament uses for
schema lookup, policy binding, and checkpoint setup.

## Snapshot resume and incremental reads

Both workflows eventually call `Resumable.ExtractFrom`, but their plans come
from different places:

* `ResumePlanner` creates a stable per-resource or per-part plan for continuing
  the same logical run. A nil checkpoint marks a resource that must be reread.
* `IncrementalPlanner` starts a later run from a durable cross-run watermark and
  the pipeline's cursor configuration.
* `CursorColumnProvider` describes eligible and recommended cursor fields. It is
  also used to bind the version field for incremental upserts.

During extraction, each row's `RowMeta.Key` identifies checkpoint progress and
the writer's part identifies which planned unit advanced. Implementations should
read only from the supplied plan; silently inventing new part layouts makes
recovery ambiguous.

## CDC

A connection is CDC when the source spec declares a CDC source policy and the
connection config sets `replication` to `cdc`. Filament derives this with
`ReplicationFor`; a source without a CDC policy is always standard. A CDC route
then uses `ChangeSource.ExtractChanges`, supplying selected resources, prior
checkpoints, limits, and an observation callback.

`ChangeAcknowledger` is narrower: after sink commit and durable checkpoint
observation, the runner can tell the source which upstream positions are safe to
release. A source must never acknowledge beyond those positions.

CDC support therefore requires agreement between connection mode, advertised
source policies, emitted ordered operations, and the extraction interface.

Continuous execution uses a reusable source session and repeated sink epochs.
Those contracts are described in
[Streaming connectors](/pages/connectors/building-a-connector/streaming),
including how to report complete source boundaries and acknowledge certified
progress.

## Validation and rate limits

Every source already implements the pure `Validate` method on `Source`.
`ConfigValidatable` provides the same pure validation hook for types that do not
otherwise require it, including sinks. It must not make network calls.

`LiveValidatable.TestConnection` performs an explicit network probe for either
a source or sink. `RateLimited.Limits` lets a source publish its own requests per
second and burst ceiling; it describes source behavior rather than replacing
run-level rate settings.

## Sink extensions

| Interface or capability | What it means today |
| - | - |
| `Schematized` | The runner calls `EnsureSchema` for each resource before extraction. Continuous runs also prepare inferred payload columns before their first batch is applied. |
| `SinkCapabilities.WritePolicies` | Declares bounded write modes. Bounded planning rejects ingestion types without a declared policy; the sink applies bound policies inside `Apply`. |
| `SinkCapabilities.Stream.WritePolicies` | Declares continuous write modes. Continuous planning selects these policies independently of the bounded list; the sink implements them through its epoch lifecycle. |
| `SinkCapabilities.EncodedIntegrity` | Every `Apply` must return an `EncodedCRC` verified over the exact serialized bytes. |
| `Transactional` | Declares stage/promote methods, but the engine does not currently call them. |

The last interface should not be confused with runtime behavior. Today the
bounded engine writes through `Sink.Apply` and finishes through `Commit` or
`Abort`. Continuous execution uses `BeginEpoch`, `CommitEpoch`, and `AbortEpoch`. A sink that declares an upsert policy implements it inside `Apply`.
A sink that needs staging manages it inside its own lifecycle.

## Keep declaration and behavior together

`ConnectorSpec.SourcePolicies` and `SinkSpec.Capabilities` are advertisements;
the interfaces are executable behavior. Review both whenever you add a mode.
Planning checks the advertised policy, write capability, primary keys, and
cursor metadata, while the runner later asserts the interfaces required to
carry it out. Keeping those two views aligned gives connector users an early,
understandable error instead of a failure after extraction begins.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.