Skip to main content
The Amazon Redshift sink (alpha) writes each resource to a typed Redshift table. It supports snapshot, incremental, and CDC pipelines on provisioned clusters and Redshift Serverless workgroups. Batches are encoded as Parquet, staged in a user-provided S3 bucket, and loaded with Redshift COPY manifests.

​
Redshift setup

The connector does not have a provisioned/Serverless switch. Supply the provisioned cluster endpoint or Serverless workgroup endpoint as host; both use the same PostgreSQL-compatible connection. The default port is 5439. The security group associated with the cluster or workgroup must allow inbound traffic from the Filament worker on the configured port. See AWS’s Redshift network configuration guide. Filament currently connects with a Redshift database username and password. The user needs permission to create and use the destination schema, create temporary tables, and create and alter destination tables. A dedicated schema owned by the Filament user is the simplest setup. Pre-existing destination tables must also allow the user to insert and delete rows.

​
S3 load staging

Create a private, general-purpose S3 bucket in the same AWS Region as the Redshift cluster or Serverless workgroup. Loading requires two distinct AWS identities:
  • The Filament worker identity uses the AWS SDK default credential chain. It needs s3:ListBucket on the bucket and s3:PutObject plus s3:DeleteObject under the configured staging prefix.
  • The Redshift associated IAM role needs s3:ListBucket and s3:GetObject for the same bucket and prefix. Associate the role with the provisioned cluster or Serverless namespace before using it for COPY.
AWS documents role creation and association in Authorizing Amazon Redshift to access AWS services. Set Associated IAM Role to default when the S3-read role is Redshift’s default role, or provide its full ARN. If role use is restricted by database user, grant the Filament user ASSUMEROLE permission for COPY.
The worker identity uploads and deletes staged objects; the associated Redshift role reads them. Granting S3 access to only one identity is not sufficient.
The default object prefix is filament/load-staging. Filament appends the pipeline ID and run-scoped hashes to prevent pipelines and runs from sharing objects:
The sink deletes staged objects after each batch on a best-effort basis. An S3 lifecycle rule on the staging prefix is recommended as a fallback. If the bucket uses SSE-KMS with a customer-managed key, the worker must be able to write with that key and the associated Redshift role must be able to decrypt it. Connection validation checks the SQL endpoint. The staging bucket is checked when a run opens on the worker because the API server and worker can use different AWS identities.

​
Configuration

The sink prefers batches no larger than 100,000 rows or approximately 256 MiB, whichever limit is reached first.

​
Supported write modes

The sink advertises full_replace, full_append, full_upsert, incremental_append, incremental_upsert, incremental_delete, cdc_append, and cdc_merge. See replication modes.
  • Replace loads a run-scoped replacement table and promotes it at commit.
  • Append adds each batch directly to the destination.
  • Upsert, delete, and merge fold repeated keys to their final operation, delete matching destination keys, and insert surviving rows in one transaction. These modes require a primary key.

​
Table management

Before extraction, EnsureSchema runs per resource:
  1. Create the destination schema and typed table if they do not exist.
  2. Inspect information_schema.columns and add missing columns.
  3. For replace, create an empty run-scoped replacement table.
Schema evolution is add-only. Existing columns are not dropped, renamed, or altered. Redshift primary-key constraints are informational; Filament uses the key fields itself when applying upserts, deletes, and CDC merges. Every Filament-created permanent table uses DISTSTYLE AUTO, SORTKEY AUTO, and ENCODE AUTO, allowing Redshift to evolve distribution, sorting, and compression based on the workload. See AWS’s automatic table optimization documentation.

​
Type mapping

​
Write mechanics and atomicity

Each Arrow batch is encoded as Snappy-compressed Parquet files. The file count is based on approximately 64 MiB of unencoded input per file and, when the row count permits, rounded up to a multiple of eight for parallel COPY work. Each manifest contains at most 128 Parquet files. Larger batches produce multiple manifests, which are copied sequentially into the same temporary table and transaction. This is a per-manifest boundary, not a row or run limit; large initial syncs continue across normal engine batches. The sink verifies S3 uploads with CRC32C, compares the loaded row count with the submitted batch, and returns an encoded checksum in the write receipt. The temporary load table, destination mutation, and replay token commit in one Redshift transaction. Append and keyed writes become visible per batch; no transaction covers an entire run. Replace batches remain in a replacement table until Commit, which locks, clears, and refills the live table in one transaction per resource. Multiple resources are promoted independently. A replay ledger named _filament_load_batches prevents the same batch from being applied twice. Reusing a replay token with different content is treated as an integrity error.

​
Failure behavior

A failed append run can leave earlier committed batches in the destination, but retrying the same run does not duplicate them. Upsert, delete, and merge retries converge through both the replay ledger and primary key. A failed replace leaves the previous destination rows intact. Abort drops the run-scoped replacement table and its replay entries. Additive schema changes made before the failure remain in place.

​
Troubleshooting

  • A connection timeout usually means the Redshift endpoint is not reachable from the worker or its associated security group does not allow the database port.
  • access bucket means the worker identity cannot inspect the configured bucket. Check the bucket name, Region, bucket policy, and s3:ListBucket.
  • S3 upload or cleanup errors require s3:PutObject and s3:DeleteObject on the configured staging prefix for the worker identity.
  • A COPY manifest access error concerns the associated Redshift role, not the worker role. Confirm its s3:GetObject access and its association with the cluster or namespace.
  • Region errors require the staging bucket and Redshift deployment to use the same AWS Region.