Skip to main content
A pipeline connects sources to destinations. Its name, description, worker settings, and schedule can change in place. Its graph cannot: changing nodes, routes, resources, or modes creates a new version. Existing runs continue to use the version on which they started. The messages are defined in protos/ingestion/v1/pipelines.proto and protos/ingestion/v1/capabilities.proto.

​
The graph

Nodes are the sources and sinks on the canvas. Each node references a connection by connection_id for the connector and connection-scoped config, and carries its own config and secret_refs. These form the pipeline-scoped overlay (target table, path prefix), shallow-merged over the connection’s config at run time with the node winning on key conflicts. Edges route data from a source node to a sink node. An edge with an empty resource routes all enabled resources. A named resource is an additive per-resource route. On standard (query-based) connections an edge carries:
  • a per-resource read_mode (FULL or INCREMENTAL)
  • a write_mode (APPEND, REPLACE, or UPSERT)
  • optional cursors: per-resource cursor column and lookback overrides, with omitted resources falling back to connector auto-detection
Standard edges default to full reads and upsert writes. Upsert uses append for resources without a primary key, while keyed resources still use upsert. The sink must support append for this fallback. Repeated keyless rows are retained. Two invariants:
  • Every edge on the same source-to-destination route must use the same write mode — the write mode belongs to the route, not the individual edge.
  • Edges over CDC connections carry no read mode. Their route-wide write mode can be APPEND or MERGE; when omitted, it defaults to APPEND.

​
RPCs

​
ValidatePipeline

Takes the graph inline, so an unsaved editor canvas validates exactly like a persisted version. Validation problems are returned in the response rather than as RPC errors. The response carries a top-level valid flag, graph-level errors (dangling node references, unknown connections), and one EdgeValidation per edge with its errors, the supported_write_modes for its route, its replication mode, and the effective_read_mode/effective_write_mode after defaulting. Each edge also reports requirements, the configuration its mode combination involves, such as a cursor column for incremental reads or a primary key for keyed writes. Keyless upsert resources report a non-blocking append fallback. A requirement carries candidate values with recommendations, satisfied (a value is configured), and blocking (the run would fail as-is). Unsatisfied but non-blocking means connector auto-detection covers it. Validation runs under a 30-second timeout.

​
CreatePipeline

Creates a pipeline from name, description, an optional inline schedule, and optional worker_configuration. Worker configuration can set Kubernetes resources, node selectors, and tolerations for this pipeline’s workers. The pipeline has no graph until the first CreatePipelineVersion.

​
CreatePipelineVersion

The only way to change what a pipeline does. The server:
  1. Runs full ValidatePipeline semantics on the submitted graph. An invalid graph is rejected with invalid_argument and nothing is written.
  2. Normalizes edge modes, making the FULL/UPSERT defaults explicit on standard edges while CDC edges stay bare.
  3. Defaults each sink node’s declared schema field (the connector’s schema_field) to the normalized name of its upstream source connection, when the node didn’t set it explicitly.
  4. Persists the version as the new current version and refreshes the schedule’s pre-created scheduled-run rows to point at it.

​
UpdatePipeline

Updates name, description, and worker_configuration without creating a new version. Worker settings live on the pipeline so changing CPU or memory does not move the pipeline to a new set of resource checkpoints. Renaming a pipeline also leaves its graph unchanged.

​
GetPipeline, ListPipelines, versions

GetPipeline and ListPipelines return lean objects by default. Opt into heavier data with include_versions, include_last_run, and include_schedule. ListPipelines also takes include_deleted and pagination. GetPipelineVersion fetches by version number, where 0 returns the current version. ListPipelineVersions returns newest first. DeletePipeline is a soft delete.

​
Schedules

A pipeline has at most one schedule. Its client-owned config:
  • cron
  • timezone: IANA name, default UTC
  • is_enabled
  • overlap_policy: SKIP (don’t fire while a scheduled run of this pipeline is still going) or ALLOW
The server owns the schedule’s identity and reports next_fire_at and last_fired_at. CreatePipelineSchedule attaches the schedule and returns already_exists if one exists. UpdatePipelineSchedule replaces the config and returns not_found if none does. There is no delete or standalone disable RPC. Pausing is UpdatePipelineSchedule with is_enabled: false, and the schedule is removed only when its pipeline is deleted. The scheduler pre-creates a SCHEDULED run row for the next fire. See Runs for how that appears in listings.

​
Notifications

A pipeline can have multiple independent notifiers, managed through CreatePipelineNotifier, UpdatePipelineNotifier, ListPipelineNotifiers, and DeletePipelineNotifier. The messages are defined in protos/ingestion/v1/notifiers.proto. The control plane and standalone application automatically deliver matching events as JSON webhooks or Slack messages. Create takes pipeline_id and a notifier input:
events lists one or more NotifierEvent values; there is no wildcard. The run lifecycle events are NOTIFIER_EVENT_RUN_STARTED, NOTIFIER_EVENT_RUN_COMPLETED, NOTIFIER_EVENT_RUN_FAILED, NOTIFIER_EVENT_RUN_PARTIAL (some resources landed, some failed), NOTIFIER_EVENT_RUN_CANCELED, and NOTIFIER_EVENT_RUN_PAUSED. Each webhook body carries the event type (for example run.failed), the run_id, and the time; run.completed adds records and bytes, and run.paused adds committed. Optional resources contains exact resource names; an empty list matches any resource. Headers are optional and can include a custom API-key header. Filament supplies Content-Type and X-Filament-* headers, so those names cannot be overridden. Notifier config follows the same split as connection config. The URL is plain configuration and is returned in config.url. Headers are the only secret: they are extracted from config, stored through the configured secrets provider, and appear in responses only as secret_refs.headers. A webhook without headers needs no secrets provider. To use a pre-existing secret, omit config.headers and supply secret_refs.headers instead; its value must be a JSON object of header names to string values. Read-only providers, including environment variables, support this reference form; inline values require a writable provider. Saving validates the destination’s format but does not contact the endpoint or verify its credentials. Update takes pipeline_id, notifier_id, and the replacement notifier settings. Send config.url and echo the secret_refs you last read. Omit config.headers to keep the current header secret, send an empty object to remove it, or send new values to replace it. The notification type cannot change. The last write wins; updating or deleting a deleted notifier returns not_found. Delete takes the same IDs. List returns all live notifiers for the pipeline, including disabled ones. Notifiers are soft-deleted with their pipeline. Replaced or deleted secrets created by this API are removed on a best-effort basis after the database change succeeds; external references are never deleted. Webhook delivery makes up to 10 attempts with exponential backoff. Delivery failures do not affect pipeline results. Restarts can cause duplicate webhooks and receivers should use X-Filament-Delivery-ID to recognize them.

​
Slack

A notifier with NOTIFICATION_TYPE_SLACK posts each matching event to a Slack incoming webhook as a Block Kit message with a status color.
The URL is a credential, so it is the secret of a Slack notifier. It is extracted from config, stored through the configured secrets provider, and appears in responses only as secret_refs.url. It is never returned. To use a pre-existing secret, omit config.url and supply secret_refs.url instead. Its value must be the URL as plain text. Inline values require a writable provider. The URL must use https and the host hooks.slack.com or hooks.slack-gov.com, with a path under /services/. On update, omit config.url to keep the stored URL or send a new one to replace it. Each message names the pipeline and the outcome, followed by the run ID and the event time. run.completed adds the record and byte counts, and run.paused adds whether the run committed. Slack notifiers take no custom headers. Delivery follows the same attempts and backoff as webhooks, and a restart can post the same message twice.