Skip to main content
The Google Cloud Storage sink (alpha) writes each resource as one NDJSON, JSON, or Parquet object per run. It uses the same run-versioned layout and success-manifest contract as the S3 sink.

​
Configuration

Authentication uses Google Application Default Credentials. On GKE, use Workload Identity; for local development, run gcloud auth application-default login before starting Filament. The sink does not require a project or region setting because a GCS bucket name is globally unique and the bucket already owns its location configuration. TestConnection reads bucket metadata. Runs require permission to create objects in the configured bucket.

​
Object layout

With prefix local_pg, resource accounts, and the default partition, a run writes:
The partition template supports {{.Date}}, {{.StartedAt}}, {{.Resource}}, and {{.Run}}. Values use the run’s UTC start time. Leave the partition empty to write objects directly under the resource directory. The success manifest lists every resource’s key, gs:// URI, row count, byte count, and CRC32C. Consumers should treat _SUCCESS.json as the visibility boundary and ignore runs without it.

​
Write mechanics and failure behavior

Each resource is streamed through a GCS resumable writer. The client buffers one configured chunk per active resource and retries transient chunk failures. Objects are finalized during Commit; the success manifest is uploaded only after all resource writers close successfully. Abort cancels unfinished resumable writers. If Commit fails after some resource objects have finalized, those objects can remain without a success manifest, matching the S3 sink’s resource-level atomicity.