Introduction

Filament is a data replication engine written in Go. It supports full loads, incremental replication, and change data capture (CDC). As data moves through the engine, Filament verifies Arrow batches at the sink handoff and, where the sink supports it, checks the exact serialized payload. You can run Filament as a standalone binary or embed it in a Go application. Connectors, the data store, and the event bus are interfaces with swappable implementations. Runs that carry resumable progress (incremental and checkpointed full reads) pick up from their last durable checkpoint instead of starting over. The requirements and recovery behavior are covered in Runs and recovery.

​

How it’s put together

The runner executes each data movement from extraction through the final sink write. The API server and control plane decide when and where that runner starts. On Kubernetes, that work is split across three processes:

• The server owns connections, pipelines, and run requests
• The control plane schedules and dispatches
• Each run executes as its own worker Job

For smaller deployments, runs can execute in the control plane, the standalone binary, or a Go process that embeds Filament. Postgres stores run state and NATS JetStream carries progress events when the components run separately. Read Architecture to follow a batch through those components.

​

Where to go next