Skip to main content
Filament runs as separate server and control-plane services backed by a datastore, an event bus, and a secret provider. Kubernetes with Helm is the recommended production deployment. Railway and Render are alternatives when Kubernetes worker Jobs are not available. DigitalOcean provides a single-process evaluation deployment.

​
Deployment options

Kubernetes, Railway, and Render run the server and control plane as separate services. The DigitalOcean template uses the standalone image to fit the platform’s one-service Deploy to DO template format.
Run exactly one control-plane replica. Multiple replicas may race while coordinating schedules and runs.

​
Service architecture

Server serves the API and web application. It accepts connection, pipeline, and run requests and persists them through the configured datastore. Control plane manages schedules and coordinates runs. With Kubernetes dispatch it creates worker Jobs and tracks their progress. With worker dispatch it leaves execution to a long-lived worker. Worker is the one process that links connector drivers. A persistent worker answers the server’s connector calls: specs, config checks, discovery, and column inspection. The same image executes runs, either as a Job per run or as a long-lived process. The server and control plane coordinate through the datastore and event bus; they do not call each other directly. Both call the persistent worker for anything that needs a connector.

​
Where runs execute

The control plane’s DISPATCH_MODE determines where runs execute:
  • kubernetes, the default, creates one worker Job per run. Each run receives its own container and resource configuration. Workers report heartbeats, and the reaper fails runs whose heartbeats become stale.
  • worker dispatches nothing. A long-lived worker started with -execute consumes requested runs from the event bus; every run shares that worker’s CPU and memory. Deployments without Kubernetes use this mode.
Both modes use the same runner; dispatch changes where it runs, not how a pipeline behaves.

​
Runtime dependencies

A Filament deployment provides three runtime dependencies:
  • Datastore persists run state, checkpoints, connections, pipelines, and schedules. The packaged server and control-plane deployment uses PostgreSQL.
  • Event bus carries run requests and progress events between runtime modules. The packaged deployment uses NATS JetStream.
  • Secret provider stores or resolves connector credentials. The default provider stores encrypted secrets in the configured datastore; AWS Secrets Manager is also supported.
The datastore-backed secret provider requires a base64-encoded AES encryption key. Provider selection and connection settings are covered in Configuration.