Skip to main content
A pipeline connects sources to destinations. Its name, description, worker settings, and schedule can change in place. Its graph cannot: changing nodes, routes, resources, or modes creates a new version. Existing runs continue to use the version on which they started. The messages are defined in protos/ingestion/v1/pipelines.proto and protos/ingestion/v1/capabilities.proto.

​
The graph

Nodes are the sources and sinks on the canvas. Each node references a connection by connection_id for the connector and connection-scoped config, and carries its own config and secret_refs. These form the pipeline-scoped overlay (target table, path prefix), shallow-merged over the connection’s config at run time with the node winning on key conflicts. Edges route data from a source node to a sink node. An edge with an empty resource routes all enabled resources. A named resource is an additive per-resource route. On standard (query-based) connections an edge carries:
  • a per-resource read_mode (FULL or INCREMENTAL)
  • a write_mode (APPEND, REPLACE, or UPSERT)
  • optional cursors: per-resource cursor column and lookback overrides, with omitted resources falling back to connector auto-detection
Two invariants:
  • Every edge on the same source-to-destination route must use the same write mode — the write mode belongs to the route, not the individual edge.
  • Edges over CDC connections carry no read mode. Their route-wide write mode can be APPEND or MERGE; when omitted, it defaults to APPEND.

​
RPCs

​
ValidatePipeline

Takes the graph inline, so an unsaved editor canvas validates exactly like a persisted version. Validation problems are returned in the response rather than as RPC errors. The response carries a top-level valid flag, graph-level errors (dangling node references, unknown connections), and one EdgeValidation per edge with its errors, the supported_write_modes for its route, its replication mode, and the effective_read_mode/effective_write_mode after defaulting. Each edge also reports requirements, the configuration its mode combination involves, such as a cursor column for incremental reads or a primary key for upserts. A requirement carries candidate values with recommendations, satisfied (a value is configured), and blocking (the run would fail as-is). Unsatisfied but non-blocking means connector auto-detection covers it. Validation runs under a 30-second timeout.

​
CreatePipeline

Creates a pipeline from name, description, an optional inline schedule, and optional worker_configuration. Worker configuration can set Kubernetes resources, node selectors, and tolerations for this pipeline’s workers. The pipeline has no graph until the first CreatePipelineVersion.

​
CreatePipelineVersion

The only way to change what a pipeline does. The server:
  1. Runs full ValidatePipeline semantics on the submitted graph. An invalid graph is rejected with invalid_argument and nothing is written.
  2. Normalizes edge modes, making the FULL/REPLACE defaults explicit on standard edges while CDC edges stay bare.
  3. Defaults each sink node’s declared schema field (the connector’s schema_field) to the normalized name of its upstream source connection, when the node didn’t set it explicitly.
  4. Persists the version as the new current version and refreshes the schedule’s pre-created scheduled-run rows to point at it.

​
UpdatePipeline

Updates name, description, and worker_configuration without creating a new version. Worker settings live on the pipeline so changing CPU or memory does not move the pipeline to a new set of resource checkpoints. Renaming a pipeline also leaves its graph unchanged.

​
GetPipeline, ListPipelines, versions

GetPipeline and ListPipelines return lean objects by default. Opt into heavier data with include_versions, include_last_run, and include_schedule. ListPipelines also takes include_deleted and pagination. GetPipelineVersion fetches by version number, where 0 returns the current version. ListPipelineVersions returns newest first. DeletePipeline is a soft delete.

​
Schedules

A pipeline has at most one schedule. Its client-owned config:
  • cron
  • timezone: IANA name, default UTC
  • is_enabled
  • overlap_policy: SKIP (don’t fire while a scheduled run of this pipeline is still going) or ALLOW
The server owns the schedule’s identity and reports next_fire_at and last_fired_at. CreatePipelineSchedule attaches the schedule and returns already_exists if one exists. UpdatePipelineSchedule replaces the config and returns not_found if none does. There is no delete or standalone disable RPC. Pausing is UpdatePipelineSchedule with is_enabled: false, and the schedule is removed only when its pipeline is deleted. The scheduler pre-creates a SCHEDULED run row for the next fire. See Runs for how that appears in listings.