Skip to main content
The Snowflake sink (alpha) writes each resource to a typed Snowflake table. It supports snapshot, incremental, and CDC pipelines using Parquet files staged through the Snowflake user’s internal stage.

Snowflake setup

Key-pair authentication is the recommended setup for the dedicated Filament service user. Filament currently accepts an unencrypted RSA private key in PKCS#8 or PKCS#1 PEM format.

1. Generate a key pair

From a secure local directory, generate a 2048-bit RSA private key and its public key:
The private key is a secret. Do not commit it or send it to Snowflake. Snowflake receives only the public key. Print the public-key body without the PEM delimiters or line breaks:
Copy that output for <PUBLIC_KEY_BODY> in the SQL below. See Snowflake’s key-pair authentication guide for key rotation and fingerprint verification.

2. Create the role, warehouse, database, and service user

Run the following as an account administrator. Change the uppercase object names if your organization uses different names.
Filament runs CREATE SCHEMA IF NOT EXISTS for the pipeline destination. A schema created by FILAMENT_ROLE is owned by that role, so it can create and evolve tables inside it. When targeting a pre-existing schema, grant at least USAGE and CREATE TABLE on that schema. Pre-existing destination tables must also permit the role to insert, update, delete, truncate, and add columns; using a Filament-owned schema avoids managing those table grants individually.

3. Configure the connection

Use these values when creating the Snowflake connection in Filament: On macOS, pbcopy < snowflake_filament_snow_key.p8 copies the complete private key while preserving its line breaks. Paste it directly into the Private Key field and validate the connection. Fields-mode connections support key-pair authentication and Snowflake programmatic access tokens. Password authentication is not exposed; Snowflake SERVICE users are intended for non-interactive authentication.

Configuration

The sink prefers batches no larger than 100,000 rows or approximately 256 MiB, whichever limit is reached first.

Supported write modes

The sink advertises full_replace, full_append, full_upsert, incremental_append, incremental_upsert, incremental_delete, cdc_append, and cdc_merge. See replication modes.
  • Replace truncates the destination before loading the snapshot.
  • Append copies each batch directly into the destination.
  • Upsert, delete, and merge load each batch into a temporary table and execute one Snowflake MERGE. These modes require a primary key.

Table management

Before extraction, EnsureSchema runs per resource:
  1. Create the destination schema and typed table if they do not exist.
  2. Truncate the table when the selected mode is replace.
  3. Inspect INFORMATION_SCHEMA.COLUMNS and add missing columns with ADD COLUMN IF NOT EXISTS.
Schema evolution is add-only. Existing columns are not dropped, renamed, or altered. Primary-key constraints are informational in Snowflake, but Filament uses the key columns to build deterministic merge conditions.

Type mapping

Write mechanics and atomicity

Each Arrow batch is encoded as a Snappy-compressed Parquet file in memory. The sink uploads it with PUT to a run-scoped path under the user’s @~ stage, loads it with COPY INTO, and removes the staged object. No external storage integration is required. Append and replace copy into the live destination. Key-based modes create a session-local temporary table, load the batch there, reduce repeated keys to their final operation, and apply the result with one MERGE transaction. Writes become visible per batch; there is no transaction covering an entire pipeline run. The sink verifies the encoded Parquet checksum before returning the write receipt. This validates the serialization boundary but does not query the committed table back from Snowflake.

Failure behavior

An append run can leave successfully loaded batches in the destination after a later failure. Upsert, delete, and merge runs can also leave completed batch merges, but retries converge through the primary key. Replace truncates before loading, and Abort truncates the replacement tables again, so a failed replace leaves those destinations empty rather than restoring their previous contents.

Troubleshooting

  • must contain an RSA private-key PEM block: paste the complete private key, including its PEM delimiters. Do not paste the .pub file.
  • encrypted PEM keys are not supported: generate the key with -nocrypt as shown above. Protect the resulting file with restrictive filesystem permissions.
  • JWT fingerprint errors: confirm that the private key matches the public key registered on FILAMENT_USER. Snowflake documents additional checks in its key-pair troubleshooting guide.
  • Schema authorization errors: verify USAGE and CREATE SCHEMA on the database, plus USAGE and CREATE TABLE on any pre-existing destination schema.