Embedded

Call app.Run to serve Filament’s API and execute runs inside your Go process. By default, it uses in-memory SQLite and an in-process event bus, so state and events disappear when the process restarts. You can replace them with Postgres and NATS for durability. The call listens on INGESTION_ADDR (default :8080) and blocks until its context is canceled or a component fails. Blank imports register only the connectors your binary needs.

Minimal embed

This compiles and runs as-is:

package main

import (
    "context"
    "log"

    "github.com/galaxy-io/filament/app"
    _ "github.com/galaxy-io/filament/connectors/postgres"
    _ "github.com/galaxy-io/filament/connectors/stdout"
)

func main() { log.Fatal(app.Run(context.Background())) }

With no options, app.Run uses an in-process event bus, in-memory SQLite, and the default connector registries populated by the blank imports.

The default in-memory SQLite database is discarded when the process exits. Use file-backed SQLite or PostgreSQL to keep state across restarts.

See the connector catalog for what is available to import, and writing a source to add your own.

Options

Every default is replaceable through an app.Option.

Option	Replaces
`WithBus`	The in-process event bus
`WithDataStore`	The in-memory SQLite datastore
`WithMetricsStore`	The dashboard metrics backend (unimplemented by default)
`WithSecrets`	The secrets provider
`WithSources`	The source registry
`WithSinks`	The sink registry
`WithUI`	The HTTP handler at `/` (e.g. `ui.Handler()`)
`WithLogger`	The logger

Whatever store you pass to WithDataStore must also implement filament.ScheduleStore, or Run returns an error — the scheduler needs it. Both the SQLite and PostgreSQL stores qualify.

Durable embed

One durable setup uses the same bus, store, and secrets as the shipped binaries:

pool, err := postgres.NewPool(ctx, dsn)          // datastore/postgres
if err != nil {
    log.Fatal(err)
}
bus, err := nats.New(natsURL, events.Codec)      // eventbus/nats
if err != nil {
    log.Fatal(err)
}
secrets, err := secretpg.NewFromEnv(pool)        // secret/postgres, reads ENCRYPTION_KEY
if err != nil {
    log.Fatal(err)
}

err = app.Run(ctx,
    app.WithBus(bus),
    app.WithDataStore(postgres.New(pool)),
    app.WithSecrets(secrets),
)

Each swap has a clear role:

eventbus/nats gives you durable, replayable JetStream delivery
datastore/postgres persists everything Filament tracks
secrets can come from secret/postgres (AES-GCM, ENCRYPTION_KEY), secret/aws, or the read-only secret/env provider

Advanced composition

When app.Run composes too much (you want only some modules, or a single run with no server), the pieces underneath are public:

module.MountAll mounts a chosen set of modules against module.Deps
eventbus/host runs them against a bus
runner.RunOne executes exactly one persisted run and exits, which is what the worker binary does

Most embedders never need this layer.

Get started

Concepts

Usage

Connectors

Deployment

API reference

Benchmarks

Contributing

Minimal embed

Options

Durable embed

Advanced composition

​Minimal embed

​Options

​Durable embed

​Advanced composition

Minimal embed

Options

Durable embed

Advanced composition