> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Cloud Storage

> Write NDJSON, JSON, or Parquet objects to Google Cloud Storage

The Google Cloud Storage sink (**alpha**) writes each resource as one NDJSON,
JSON, or Parquet object per run. It uses the same run-versioned layout and
success-manifest contract as the S3 sink.

## Configuration

| Field | Scope | Default | Description |
| - | - | - | - |
| `bucket` | Connection | — | Required. Destination GCS bucket. |
| `auth_method` | Connection | `application_default_credentials` | Uses the attached service account, GKE Workload Identity, or local ADC |
| `prefix` | Pipeline | — | Root folder for the pipeline, with surrounding `/` trimmed |
| `partition` | Pipeline | `dt={{.Date}}` | Folders between each resource and its files |
| `file_format` | Pipeline | `ndjson` | `ndjson`, `jsonl`, `json`, or `parquet` |
| `compression` | Pipeline | per format | `none` or `gzip` for JSON formats; `snappy` for Parquet |
| `chunk_size_mib` | Pipeline | `16` | Resumable-upload chunk size in MiB, from 1 through 1024 |
| `upload_concurrency` | Pipeline | `4` | Concurrent GCS operations across resources, from 1 through 32 |

Authentication uses Google Application Default Credentials. On GKE, use
Workload Identity; for local development, run
`gcloud auth application-default login` before starting Filament. The sink
does not require a project or region setting because a GCS bucket name is
globally unique and the bucket already owns its location configuration.

`TestConnection` reads bucket metadata. Runs require permission to create
objects in the configured bucket.

## Object layout

With prefix `local_pg`, resource `accounts`, and the default partition, a run
writes:

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
local_pg/accounts/dt=2026-09-10/<run>.parquet
local_pg/_runs/<run>/_SUCCESS.json
```

The partition template supports `{{.Date}}`, `{{.StartedAt}}`,
`{{.Resource}}`, and `{{.Run}}`. Values use the run's UTC start time. Leave the
partition empty to write objects directly under the resource directory.

The success manifest lists every resource's key, `gs://` URI, row count, byte
count, and CRC32C. Consumers should treat `_SUCCESS.json` as the visibility
boundary and ignore runs without it.

## Write mechanics and failure behavior

Each resource is streamed through a GCS resumable writer. The client buffers
one configured chunk per active resource and retries transient chunk failures.
Objects are finalized during `Commit`; the success manifest is uploaded only
after all resource writers close successfully.

`Abort` cancels unfinished resumable writers. If `Commit` fails after some
resource objects have finalized, those objects can remain without a success
manifest, matching the S3 sink's resource-level atomicity.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.