Skip to main content
The Amazon S3 sink (alpha) writes each resource as one NDJSON, JSON, or Parquet object per run in an S3-compatible bucket, laid out so an external table per resource can read it directly. It is not a lakehouse format with table-level atomic commits; see the Iceberg sink for that.

​
Configuration

A prefix left empty is defaulted server-side to the normalized source connection name. Instance Profile is the default. It uses the AWS SDK default credential chain, including EC2 instance profiles and container or workload roles. Choose IAM Credentials to provide access_key_id and secret_access_key; add session_token when using temporary credentials. TestConnection performs a read-only HeadBucket request. On AWS this requires s3:ListBucket; it does not prove that the identity can write. Runs also need object-write and multipart-upload permissions for the configured prefix.

​
Supported write modes

The sink advertises full_replace and full_append. See replication modes. Because every run writes its own objects, the two modes behave identically here. Each run produces a fresh object per resource and never touches a previous run’s objects. A full replace run is therefore a snapshot in its own date partition. Query the latest partition to read the current state, or every partition to read history.

​
Object layout

Each successful run writes one object per resource and a success manifest. With prefix local_pg, resources accounts and orders, and the default partition, a run looks like:
The path is prefix, resource, partition folders, then the run identifier as the filename. Objects are resource-major so one external table per resource can point at local_pg/accounts/. The resource folder, the run identifier filename, and the manifest location are fixed. Only the partition folders are yours to choose. Manifests live under _runs, which query engines skip because of the leading underscore. Object files carry the run identifier so runs on the same day never collide.

​
Partition template

partition sets the folders between each resource and its files. Write it as a path with placeholders in double braces. Every placeholder is filled from the run’s start time in UTC, so every object of one run lands in the same partition. The default dt={{.Date}} is a Hive-style partition that Athena, Glue crawlers, Spark, and DuckDB discover without extra configuration. Other layouts: Format takes a Go time layout, where 2006, 01, 02, and 15 stand for year, month, day, and hour. Plain directories without key=value names need Athena partition projection with a location template. The template is checked when the pipeline is saved. It must produce a relative path with no empty, ., or .. segments, and unknown placeholders are rejected. Helm charts that render pipeline YAML treat {{ as their own syntax, so escape the template there as {{ "{{" }}.Date{{ "}}" }}.

​
File formats

file_format chooses how each resource object is encoded. The extension and Content-Type follow the format, and gzip adds a .gz suffix and Content-Encoding: gzip. JSON formats write source field names and JSON-compatible values:
Parquet writes typed columns from the resource schema, so a selected resource with no rows still produces a valid file with the right columns. Parquet is the format to choose when the bucket is queried by Athena, Spark, or DuckDB. The compression field shows only the options valid for the selected format. The success manifest lists every resource’s key, URI, rows, bytes, and object CRC32C. Programmatic consumers should treat _SUCCESS.json as the visibility boundary and ignore runs without it. Query engines cannot apply that rule and read every object under the table location; see failure behavior below.

​
Write mechanics and atomicity

The sink streams full part_size_mib buffers as multipart uploads while the run is active. upload_concurrency limits S3 requests across resources; when that limit is reached, source extraction waits. At commit, Filament uploads the remaining bytes and completes each object. Small resources use PutObject, and a selected resource with no rows becomes a zero-byte object. The sink does not need temporary disk for the full run. Memory use comes mainly from unfinished multipart buffers, in-flight uploads, and the current encoded batch. Objects smaller than one part remain in memory until commit. Each resource object is atomic, but a group of S3 objects cannot be committed atomically. Resource objects may be visible while Commit is still running; the run becomes valid only when _SUCCESS.json is uploaded last.

​
Failure behavior

Abort cancels in-flight work and abandons incomplete multipart uploads. If Commit fails after some resources complete, those objects remain in their resource partitions, but no _SUCCESS.json is published, so the run is not valid. Query engines still read those objects, so remove them by run identifier before rerunning if partial data matters. Configure an S3 lifecycle rule to expire incomplete multipart uploads. Multipart uploads do not resume across worker attempts; resource progress is saved only after commit succeeds.