Skip to main content
A connection stores the settings needed to reach a source or destination. A pipeline can reuse that connection instead of repeating a hostname, bucket, or credentials on every node. Connections belong to one tenant, use version to prevent conflicting edits, and are soft-deleted. The messages are defined in protos/ingestion/v1/connections.proto.

​
Config scoping

Every field in a connector’s ConfigSchema carries a FieldScope that decides where its value lives:
  • Connection-scoped fields (FIELD_SCOPE_CONNECTION), such as host, port, and credentials, are stored on the connection. They are what these RPCs manage.
  • Pipeline-scoped fields (FIELD_SCOPE_PIPELINE), such as target table and path prefix, are stored on the pipeline node that references the connection, and shallow-merged over the connection’s config at run time, with the node winning on key conflicts.
Fields marked secret in the schema get special handling. On create and update the server extracts their values from config, stores them through the configured secrets provider, and keeps only a reference in secret_refs. Reads return those fields masked, and the API never echoes a secret back. A client can also supply secret_refs directly to point at pre-existing secrets.

​
RPCs

​
CreateConnection

Takes kind, name, connector, config, and optional secret_refs. The server resolves referenced secrets, validates the resulting values against the connector’s schema, and runs the connector’s configuration checks. These are local checks: saving a connection does not prove that the external system is reachable or that its credentials work. A secret reference that cannot be resolved returns failed_precondition.

​
UpdateConnection

Takes the full Connection object, including id and the version you last read. Semantics:
  • A version that no longer matches the stored row returns aborted. Re-read the connection and retry with the fresh version.
  • connector and kind are immutable, and so is the replication mode a source’s config implies (standard vs CDC). Changing any of them returns invalid_argument, so create a new connection instead.
  • Updating a deleted connection returns failed_precondition.
Masked secret values round-trip safely. Submit the masked config unchanged and the stored secrets stay in place. Submit a new plaintext value and it replaces the stored secret.

​
GetConnection

Returns the connection with secret fields masked. Sources additionally report replication, the effective replication mode derived from the stored config. Sinks report REPLICATION_MODE_UNSPECIFIED.

​
ListConnections

Filters by kind (unspecified returns both sources and sinks) and supports pagination. Deleted connections are excluded unless include_deleted is set.

​
DeleteConnection

This is a soft delete. The row is stamped deleted_at and its stored secret references are removed, but history referring to it stays readable. The call refuses with failed_precondition while any pipeline’s current version references the connection — remove it from those pipelines (by creating new versions without it) first. Only the current version counts. Older versions that reference it do not block deletion.