Skip to main content
Connector ID gcs The Google Cloud Storage sink (alpha) writes each resource as one NDJSON, JSON, or Parquet object per finite write. It uses the same versioned layout and success-manifest contract as the S3 sink.

​
Configuration

Authentication uses Google Application Default Credentials. On GKE, use Workload Identity; for local development, run gcloud auth application-default login before starting Filament. The sink does not require a project or region setting because a GCS bucket name is globally unique and the bucket already owns its location configuration. TestConnection reads bucket metadata. Runs require permission to create objects in the configured bucket.

​
Object layout

With prefix local_pg, resource accounts, and the default partition, a bounded run writes the following objects.
The partition template supports {{.Date}}, {{.StartedAt}}, {{.Resource}}, and {{.Run}}. Bounded date values use the run’s UTC start time. Continuous date values use each finite write’s UTC start time, while {{.Run}} retains the logical run ID. Leave the partition empty to write objects directly under the resource directory. Continuous append creates a fresh sink per boundary and uses this layout.
Each boundary has its own objects and manifest. The manifest retains the logical run ID. Continuous replace is not supported. The success manifest lists every resource’s key, gs:// URI, row count, byte count, and CRC32C. Consumers should treat _SUCCESS.json as the visibility boundary and ignore writes without it.

​
Write mechanics and failure behavior

Each resource is streamed through a GCS resumable writer. The client buffers one configured chunk per active resource and retries transient chunk failures. Objects are finalized during Commit; the success manifest is uploaded only after all resource writers close successfully. Abort cancels unfinished resumable writers. If Commit fails after some resource objects have finalized, those objects can remain without a success manifest, matching the S3 sink’s resource-level atomicity. See Continuous streaming runs for boundary defaults, setup costs, and checkpoint recovery. A replay after publication can produce another append object with the same rows under a new write identity.