Snowflake setup
Key-pair authentication is the recommended setup for the dedicated Filament service user. Filament currently accepts an unencrypted RSA private key in PKCS#8 or PKCS#1 PEM format.1. Generate a key pair
From a secure local directory, generate a 2048-bit RSA private key and its public key:<PUBLIC_KEY_BODY> in the SQL below. See Snowflake’s
key-pair authentication guide
for key rotation and fingerprint verification.
2. Create the role, warehouse, database, and service user
Run the following as an account administrator. Change the uppercase object names if your organization uses different names.CREATE SCHEMA IF NOT EXISTS for the pipeline destination. A
schema created by FILAMENT_ROLE is owned by that role, so it can create and
evolve tables inside it. When targeting a pre-existing schema, grant at least
USAGE and CREATE TABLE on that schema. Pre-existing destination tables must
also permit the role to insert, update, delete, truncate, and add columns; using
a Filament-owned schema avoids managing those table grants individually.
3. Configure the connection
Use these values when creating the Snowflake connection in Filament:
On macOS,
pbcopy < snowflake_filament_snow_key.p8 copies the complete private
key while preserving its line breaks. Paste it directly into the Private Key
field and validate the connection.
Fields-mode connections support key-pair authentication and Snowflake
programmatic access tokens. Password authentication is not exposed; Snowflake
SERVICE users are intended for non-interactive authentication.
Configuration
The sink prefers batches no larger than 100,000 rows or approximately 256 MiB,
whichever limit is reached first.
Supported write modes
The sink advertisesfull_replace, full_append, full_upsert,
incremental_append, incremental_upsert, incremental_delete, cdc_append,
and cdc_merge. See
replication modes.
- Replace truncates the destination before loading the snapshot.
- Append copies each batch directly into the destination.
- Upsert, delete, and merge load each batch into a temporary table and execute
one Snowflake
MERGE. These modes require a primary key.
Table management
Before extraction,EnsureSchema runs per resource:
- Create the destination schema and typed table if they do not exist.
- Truncate the table when the selected mode is replace.
- Inspect
INFORMATION_SCHEMA.COLUMNSand add missing columns withADD COLUMN IF NOT EXISTS.
Type mapping
Write mechanics and atomicity
Each Arrow batch is encoded as a Snappy-compressed Parquet file in memory. The sink uploads it withPUT to a run-scoped path under the user’s @~ stage,
loads it with COPY INTO, and removes the staged object. No external storage
integration is required.
Append and replace copy into the live destination. Key-based modes create a
session-local temporary table, load the batch there, reduce repeated keys to
their final operation, and apply the result with one MERGE transaction.
Writes become visible per batch; there is no transaction covering an entire
pipeline run.
The sink verifies the encoded Parquet checksum before returning the write
receipt. This validates the serialization boundary but does not query the
committed table back from Snowflake.
Failure behavior
An append run can leave successfully loaded batches in the destination after a later failure. Upsert, delete, and merge runs can also leave completed batch merges, but retries converge through the primary key. Replace truncates before loading, andAbort truncates the replacement tables again, so a failed replace
leaves those destinations empty rather than restoring their previous contents.
Troubleshooting
must contain an RSA private-key PEM block: paste the complete private key, including its PEM delimiters. Do not paste the.pubfile.encrypted PEM keys are not supported: generate the key with-nocryptas shown above. Protect the resulting file with restrictive filesystem permissions.- JWT fingerprint errors: confirm that the private key matches the public key
registered on
FILAMENT_USER. Snowflake documents additional checks in its key-pair troubleshooting guide. - Schema authorization errors: verify
USAGEandCREATE SCHEMAon the database, plusUSAGEandCREATE TABLEon any pre-existing destination schema.