What is Filament?

Filament is an open-source data movement engine built by Galaxy. It replicates and ingests data from databases, HTTP APIs, event streams, and object storage into warehouses, lakes, and databases. It supports full loads, incremental replication, and change data capture (CDC). Filament is written in Go and licensed under Apache 2.0, and the full engine is developed in the open on GitHub. As data moves through the engine, Filament verifies Arrow batches at the sink handoff and, where the sink supports it, checks the exact serialized payload. You can run Filament as a standalone binary, deploy it on Kubernetes, or embed it in a Go application. For published throughput, wall time, and memory results against other replication tools, see Benchmarking. Connectors, the data store, and the event bus are interfaces with swappable implementations. Runs that carry resumable progress (incremental and checkpointed full reads) pick up from their last durable checkpoint instead of starting over. The requirements and recovery behavior are covered in Runs and recovery.

​

How it’s put together

The runner executes each data movement from extraction through the final sink write. The API server and control plane decide when and where that runner starts. On Kubernetes, that work is split across three processes:

• The server owns connections, pipelines, and run requests
• The control plane schedules and dispatches
• Each run executes as its own worker Job

For smaller deployments, runs can execute in the control plane, the standalone binary, or a Go process that embeds Filament. Postgres stores run state and NATS JetStream carries progress events when the components run separately. Read Architecture to follow a batch through those components.

​

Where to go next