> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines

> Define data routes, save versioned changes, and schedule runs

A pipeline connects sources to destinations. Its name, description, worker
settings, and schedule can change in place. Its graph cannot: changing nodes,
routes, resources, or modes creates a new version. Existing runs continue to
use the version on which they started. The messages are defined in
`protos/ingestion/v1/pipelines.proto` and
`protos/ingestion/v1/capabilities.proto`.

## The graph

**Nodes** are the sources and sinks on the canvas. Each node references a
[connection](/pages/api-reference/service/connections) by `connection_id` for
the connector and connection-scoped config, and carries its own `config` and
`secret_refs`. These form the pipeline-scoped overlay (target table, path
prefix), shallow-merged over the connection's config at run time with the node
winning on key conflicts.

**Edges** route data from a source node to a sink node. An edge with an empty
`resource` routes all enabled resources. A named `resource` is an additive
per-resource route. On standard (query-based) connections an edge carries:

* a per-resource `read_mode` (`FULL` or `INCREMENTAL`)
* a `write_mode` (`APPEND`, `REPLACE`, or `UPSERT`)
* optional `cursors`: per-resource cursor column and lookback overrides, with
  omitted resources falling back to connector auto-detection

Standard edges default to full reads and upsert writes. Upsert uses append for
resources without a primary key, while keyed resources still use upsert. The
sink must support append for this fallback. Repeated keyless rows are retained.

Two invariants:

* Every edge on the same source-to-destination route must use the same write
  mode — the write mode belongs to the route, not the individual edge.
* Edges over CDC connections carry no read mode. Their route-wide write mode
  can be `APPEND` or `MERGE`; when omitted, it defaults to `APPEND`.

## RPCs

| RPC | Description |
| - | - |
| `ValidatePipeline` | Validate an inline, unsaved graph edge by edge |
| `CreatePipeline` | Create the pipeline's metadata shell |
| `CreatePipelineVersion` | Validate and persist a new immutable graph version |
| `UpdatePipeline` | Update metadata in place. Never creates a version |
| `GetPipeline` | Fetch a pipeline with optional expansions |
| `GetPipelineVersion` | Fetch one version (`0` means current) |
| `ListPipelineVersions` | List a pipeline's versions, newest first |
| `ListPipelines` | List pipelines with the same expansion flags |
| `DeletePipeline` | Soft-delete a pipeline |
| `CreatePipelineSchedule` | Attach the pipeline's one schedule |
| `UpdatePipelineSchedule` | Replace the schedule's configuration |

## ValidatePipeline

Takes the graph inline, so an unsaved editor canvas validates exactly like a
persisted version. Validation problems are returned in the response rather
than as RPC errors. The response carries a
top-level `valid` flag, graph-level `errors` (dangling node references, unknown
connections), and one `EdgeValidation` per edge with its errors, the
`supported_write_modes` for its route, its `replication` mode, and the
`effective_read_mode`/`effective_write_mode` after defaulting.

Each edge also reports `requirements`, the configuration its mode combination
involves, such as a cursor column for incremental reads or a primary key for
keyed writes. Keyless upsert resources report a non-blocking append fallback.
A requirement carries candidate values with recommendations,
`satisfied` (a value is configured), and `blocking` (the run would fail
as-is). Unsatisfied but non-blocking means connector auto-detection covers
it. Validation runs under a 30-second timeout.

## CreatePipeline

Creates a pipeline from `name`, `description`, an optional inline `schedule`,
and optional `worker_configuration`. Worker configuration can set Kubernetes
resources, node selectors, and tolerations for this pipeline's workers. The
pipeline has no graph until the first `CreatePipelineVersion`.

## CreatePipelineVersion

The only way to change what a pipeline does. The server:

1. Runs full `ValidatePipeline` semantics on the submitted graph. An invalid
   graph is rejected with `invalid_argument` and nothing is written.
2. Normalizes edge modes, making the `FULL`/`UPSERT` defaults explicit on
   standard edges while CDC edges stay bare.
3. Defaults each sink node's declared schema field (the connector's
   `schema_field`) to the normalized name of its upstream source connection,
   when the node didn't set it explicitly.
4. Persists the version as the new current version and refreshes the
   schedule's pre-created scheduled-run rows to point at it.

## UpdatePipeline

Updates `name`, `description`, and `worker_configuration` without creating a
new version. Worker settings live on the pipeline so changing CPU or memory
does not move the pipeline to a new set of resource checkpoints. Renaming a
pipeline also leaves its graph unchanged.

## GetPipeline, ListPipelines, versions

`GetPipeline` and `ListPipelines` return lean objects by default. Opt into
heavier data with `include_versions`, `include_last_run`, and
`include_schedule`. `ListPipelines` also takes `include_deleted` and
[pagination](/pages/api-reference/service/introduction#pagination).
`GetPipelineVersion` fetches by version number, where `0` returns the current
version. `ListPipelineVersions` returns newest first. `DeletePipeline` is a
soft delete.

## Schedules

A pipeline has at most one schedule. Its client-owned config:

* `cron`
* `timezone`: IANA name, default UTC
* `is_enabled`
* `overlap_policy`: `SKIP` (don't fire while a scheduled run of this pipeline
  is still going) or `ALLOW`

The server owns the schedule's identity and reports `next_fire_at`
and `last_fired_at`.

`CreatePipelineSchedule` attaches the schedule and returns `already_exists`
if one exists. `UpdatePipelineSchedule` replaces the config and returns
`not_found` if none does. There is no delete or standalone disable RPC.
Pausing is `UpdatePipelineSchedule` with `is_enabled: false`, and the
schedule is removed only when its pipeline is deleted. The scheduler
pre-creates a `SCHEDULED` run row for the next fire. See
[Runs](/pages/api-reference/service/runs#listruns) for how that appears in
listings.

## Notifications

A pipeline can have multiple independent notifiers, managed through
`CreatePipelineNotifier`, `UpdatePipelineNotifier`, `ListPipelineNotifiers`,
and `DeletePipelineNotifier`. The messages are defined in
`protos/ingestion/v1/notifiers.proto`.

The control plane and standalone application automatically deliver matching
events as JSON webhooks or Slack messages.

Create takes `pipeline_id` and a `notifier` input:

```json theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "pipelineId": "<pipeline-id>",
  "notifier": {
    "name": "Pipeline notifications",
    "notificationType": "NOTIFICATION_TYPE_WEBHOOK",
    "isEnabled": true,
    "events": ["NOTIFIER_EVENT_RUN_COMPLETED", "NOTIFIER_EVENT_RUN_FAILED"],
    "config": {
      "url": "https://example.com/webhook",
      "headers": {"Authorization": "Bearer <token>"}
    }
  }
}
```

`events` lists one or more `NotifierEvent` values; there is no wildcard. The
run lifecycle events are `NOTIFIER_EVENT_RUN_STARTED`,
`NOTIFIER_EVENT_RUN_COMPLETED`, `NOTIFIER_EVENT_RUN_FAILED`,
`NOTIFIER_EVENT_RUN_PARTIAL` (some resources landed, some failed),
`NOTIFIER_EVENT_RUN_CANCELED`, and `NOTIFIER_EVENT_RUN_PAUSED`. Each webhook
body carries the event `type` (for example `run.failed`), the `run_id`, and the
time; `run.completed` adds `records` and `bytes`, and `run.paused` adds
`committed`. Optional `resources` contains exact resource names; an empty list
matches any resource. Headers are optional and can include a custom API-key header.
Filament supplies `Content-Type` and `X-Filament-*` headers, so those names
cannot be overridden.

Notifier config follows the same split as connection config. The URL is plain
configuration and is returned in `config.url`. Headers are the only secret:
they are extracted from `config`, stored through the configured secrets
provider, and appear in responses only as `secret_refs.headers`. A webhook
without headers needs no secrets provider. To use a pre-existing secret, omit
`config.headers` and supply `secret_refs.headers` instead; its value must be a
JSON object of header names to string values. Read-only providers, including
environment variables, support this reference form; inline values require a
writable provider. Saving validates the destination's format but does not
contact the endpoint or verify its credentials.

Update takes `pipeline_id`, `notifier_id`, and the replacement `notifier`
settings. Send `config.url` and echo the `secret_refs` you last read. Omit
`config.headers` to keep the current header secret, send an empty object to
remove it, or send new values to replace it. The notification type cannot
change. The last write wins; updating or deleting a deleted notifier returns
`not_found`. Delete takes the same IDs.

List returns all live notifiers for the pipeline, including disabled ones.
Notifiers are soft-deleted with their pipeline. Replaced or deleted secrets
created by this API are removed on a best-effort basis after the database
change succeeds; external references are never deleted.

Webhook delivery makes up to 10 attempts with exponential backoff. Delivery
failures do not affect pipeline results. Restarts can cause duplicate webhooks
and receivers should use `X-Filament-Delivery-ID` to recognize them.

### Slack

A notifier with `NOTIFICATION_TYPE_SLACK` posts each matching event to a Slack
incoming webhook as a Block Kit message with a status color.

```json theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "pipelineId": "<pipeline-id>",
  "notifier": {
    "name": "Pipeline alerts",
    "notificationType": "NOTIFICATION_TYPE_SLACK",
    "isEnabled": true,
    "events": ["NOTIFIER_EVENT_RUN_FAILED"],
    "config": {
      "url": "https://hooks.slack.com/services/<path>"
    }
  }
}
```

The URL is a credential, so it is the secret of a Slack notifier. It is
extracted from `config`, stored through the configured secrets provider, and
appears in responses only as `secret_refs.url`. It is never returned. To use a
pre-existing secret, omit `config.url` and supply `secret_refs.url` instead.
Its value must be the URL as plain text. Inline values require a writable
provider.

The URL must use `https` and the host `hooks.slack.com` or
`hooks.slack-gov.com`, with a path under `/services/`. On update, omit
`config.url` to keep the stored URL or send a new one to replace it.

Each message names the pipeline and the outcome, followed by the run ID and
the event time. `run.completed` adds the record and byte counts, and
`run.paused` adds whether the run committed. Slack notifiers take no custom
headers. Delivery follows the same attempts and backoff as webhooks, and a
restart can post the same message twice.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.