filament CLI, which uses a YAML configuration
file.
Shared configuration
The server, control plane, and worker read the variables in this section. Only the worker links connector drivers; the server and control plane reach them throughWORKER_URL.
Private HTTP manifests
Set it on every worker: the persistent worker that answers connector calls and
the workers that execute runs. Files are loaded once at startup; missing,
empty, or invalid configured directories stop startup. The variable does not mount or distribute files. See
private catalog setup
for the directory structure and override behavior.
Connector allowlist
Set them on every worker, the same as
HTTP_MANIFESTS_DIR. The server and
control plane learn the catalog from the worker, so connectors left off the
list do not appear in the UI or CLI, and runs that use them fail. Names are
unversioned. Listing an HTTP connector such as instantly keeps its default
alias and every registered version. An unknown or versioned name stops
startup, and so does a value such as , that names no connectors. The Helm chart sets these from worker.connectors.sources and
worker.connectors.sinks.
Datastore
Use PostgreSQL for a split deployment. The service binaries accept SQLite,
but a local SQLite file is not a shared datastore for services running on
different hosts.
Event bus
DISPATCH_MODE changes where runs execute; it does not replace the event bus.Secret provider
The datastore-backed provider stores AES-GCM encrypted values in PostgreSQL or
SQLite. Every service must receive the same encryption key.
The AWS provider loads credentials through the standard AWS SDK chain. It
stores each Filament secret as a JSON envelope in AWS Secrets Manager.
The GCP provider loads Application Default Credentials and uses regional Secret
Manager secrets. Filament refs are mapped to deterministic
GCP-safe IDs and retained in secret annotations. Grant workloads access through
their attached service account or Workload Identity. Deleting a connection
permanently deletes its managed GCP secrets.
Managed OAuth refresh is supported with all four providers. Access tokens and
expiry stay in memory for one run. Only a changed refresh token is written back
to its existing secret. Rotation never creates a secret or updates a connection
row. PostgreSQL and SQLite update the encrypted value in place. AWS uses
PutSecretValue.
GCP adds a version and then
destroys the previous version.
Grant GCP workloads the
secretmanager.versions.destroy permission as well as
read and write access. Cloud providers do not require ENCRYPTION_KEY.
Refreshes are serialized within each run. Separate runs have independent token
state and do not acquire datastore leases. Avoid overlapping runs that share an
authorization when the upstream provider invalidates refresh tokens on use.
The datastore no longer tracks cloud secret versions or refresh ownership.
A failed save stops the read. The run retains a successful exchange in memory
for another save attempt. A process exit between upstream rotation and persistence
can require fresh authorization. If GCP version cleanup fails, rotation reports
an error even though the new version is already available. The previous version
may need cleanup.
Logs and OpenTelemetry
Filament enables OpenTelemetry export when any OTLP endpoint is set. It
creates both metrics and trace exporters using the selected protocol. The
exporters also honor the standard OTLP headers, TLS, timeout, and compression
variables for that transport. Application logs remain JSON on standard output;
Filament does not export logs through OTLP.
Server
Authentication
Only the server reads authentication variables.
Control plane
In
kubernetes mode, the control plane creates a worker Job for each run and
uses the reaper to detect workers that stop reporting heartbeats. In worker
mode, the control plane dispatches nothing: a long-lived worker started with
-execute consumes requested runs from the event bus. There are no worker
Jobs, heartbeats, or reaper.
Pipeline webhooks run automatically in the control plane and standalone app,
using the existing datastore and secrets provider. Standalone uses environment
secret references for destinations. See Notifications.
Kubernetes dispatch
These variables apply only whenDISPATCH_MODE=kubernetes. The Helm chart
sets them from its worker values.
Worker
The worker is the one binary that links connector drivers. Without flags it executes one dispatched run and exits. With-serve it runs long-lived and
answers connector calls for the server and control plane, needing neither
datastore nor event bus. With -execute it runs long-lived and executes
requested runs from the event bus, for deployments without Kubernetes. The two
flags combine; neither combines with RUN_ID.
With -serve, /readyz reports whether connector serving is available. This
also applies to -serve -execute: a datastore or event-bus outage does not remove
the connector service from its Service endpoints. An execution-only worker’s
readiness checks its datastore and event bus. Both modes report not ready before
startup completes and during shutdown.
The Kubernetes dispatcher injects
TENANT_ID and RUN_ID. The worker loads
the remaining shared settings from its Secret and ConfigMap. Invalid or
non-positive HEARTBEAT_SECONDS values use the 30-second default.
Standalone
Standalone is intended for testing and evaluation. It has its own embedded NATS server, SQLite datastore, and read-only environment secret provider.
Standalone also reads
LOG_LEVEL and NATS_TTL_SECONDS. Its secret provider
maps a reference such as postgres-creds/dsn to POSTGRES_CREDS_DSN. It
cannot create, update, or delete secrets.
The public Go API also uses INGESTION_ADDR. See
Embedded for library configuration.
Migrations
PostgreSQL migrations are embedded in the server image. The server’s-migrate flag waits up to five minutes for the database, applies pending
migrations, and exits: