Skip to main content
Connector ID s3 The Amazon S3 sink (alpha) writes each resource as one NDJSON, JSON, or Parquet object per finite write in an S3-compatible bucket, laid out so an external table per resource can read it directly. It is not a lakehouse format with table-level atomic commits. See the Iceberg sink for that.

​
Configuration

A prefix left empty is defaulted server-side to the normalized source connection name. Instance Profile is the default. It uses the AWS SDK default credential chain, including EC2 instance profiles and container or workload roles. Choose IAM Credentials to provide access_key_id and secret_access_key; add session_token when using temporary credentials. TestConnection performs a read-only HeadBucket request. On AWS this requires s3:ListBucket; it does not prove that the identity can write. Runs also need object-write and multipart-upload permissions for the configured prefix.

​
Supported write modes

The sink advertises full_replace and full_append. See replication modes. Because every run writes its own objects, the two modes behave identically here. Each run produces a fresh object per resource and never touches a previous run’s objects. A full replace run is therefore a snapshot in its own date partition. Query the latest partition to read the current state, or every partition to read history.

​
Object layout

Each successful bounded run writes one object per resource and a success manifest. With prefix local_pg, resources accounts and orders, and the default partition, a bounded run has this layout.
The bounded path contains the prefix, resource, partition folders, and run ID. Continuous append uses a fresh write identity for every boundary while retaining the logical run ID in the manifest. A continuous boundary has this layout.
Resource objects remain grouped by resource, so an external table can point at local_pg/accounts/. Manifests live under _runs, which query engines usually skip. Continuous replace is not supported. Each append boundary reopens the sink and produces independent objects and a manifest.

​
Partition template

partition sets the folders between each resource and its files. Write it as a path with placeholders in double braces. Time placeholders use the bounded run’s start time or the continuous write’s start time in UTC. Every object in that finite write uses the same time. The logical run ID in {{.Run}} stays unchanged across continuous boundaries. The default dt={{.Date}} is a Hive-style partition that Athena, Glue crawlers, Spark, and DuckDB discover without extra configuration. Other layouts: Format takes a Go time layout, where 2006, 01, 02, and 15 stand for year, month, day, and hour. Plain directories without key=value names need Athena partition projection with a location template. The template is checked when the pipeline is saved. It must produce a relative path with no empty, ., or .. segments, and unknown placeholders are rejected. Helm charts that render pipeline YAML treat {{ as their own syntax, so escape the template there as {{ "{{" }}.Date{{ "}}" }}.

​
File formats

file_format chooses how each resource object is encoded. The extension and Content-Type follow the format, and gzip adds a .gz suffix and Content-Encoding: gzip. JSON formats write source field names and JSON-compatible values:
Parquet writes typed columns from the resource schema, so a selected resource with no rows still produces a valid file with the right columns. Parquet is the format to choose when the bucket is queried by Athena, Spark, or DuckDB. The compression field shows only the options valid for the selected format. The success manifest lists every resource’s key, URI, rows, bytes, and object CRC32C. Programmatic consumers should treat _SUCCESS.json as the visibility boundary and ignore writes without it. Query engines cannot apply that rule and read every object under the table location. See failure behavior below.

​
Write mechanics and atomicity

The sink streams full part_size_mib buffers as multipart uploads while the run is active. upload_concurrency limits S3 requests across resources; when that limit is reached, source extraction waits. At commit, Filament uploads the remaining bytes and completes each object. Small resources use PutObject, and a selected resource with no rows becomes a zero-byte object. The sink does not need temporary disk for the full run. Memory use comes mainly from unfinished multipart buffers, in-flight uploads, and the current encoded batch. Objects smaller than one part remain in memory until commit. Each resource object is atomic, but a group of S3 objects cannot be committed atomically. Resource objects may be visible while Commit is still running. The write becomes valid only when _SUCCESS.json is uploaded last.

​
Failure behavior

Abort cancels in-flight work and abandons incomplete multipart uploads. If Commit fails after some resources complete, those objects remain in their resource partitions, but no _SUCCESS.json is published, so the write is not valid. Query engines still read those objects. If partial data matters, remove them by their run ID for bounded output or write ID for continuous output before retrying. Configure an S3 lifecycle rule to expire incomplete multipart uploads. Multipart uploads do not resume across worker attempts. Resource progress is saved only after commit succeeds. See Continuous streaming runs for boundary defaults, setup costs, and checkpoint recovery. A replay after publication can produce another append object with the same rows under a new write identity.