Filament is a data replication engine written in Go. It supports full loads,
incremental replication, and change data capture (CDC). As data moves through
the engine, Filament verifies Arrow batches at the sink handoff and, where the
sink supports it, checks the exact serialized payload. You can run Filament as
a standalone binary or embed it in a Go application.Connectors, the data store, and the event bus are interfaces with
swappable implementations.Runs that carry resumable progress (incremental and checkpointed full reads)
pick up from their last durable checkpoint instead of starting over. The
requirements and recovery behavior are covered in
Runs and recovery.
The runner executes each data movement from extraction through the final sink
write. The API server and control plane decide when and where that runner
starts.On Kubernetes, that work is split across three processes:
The server owns connections, pipelines, and run requests
The control plane schedules and dispatches
Each run executes as its own worker Job
For smaller deployments, runs can execute in the control plane, the standalone
binary, or a Go process that embeds Filament. Postgres stores run state and
NATS JetStream carries progress events when the components run separately.Read Architecture to follow a batch
through those components.
