Skip to main content
The ClickHouse sink (alpha) writes each resource to a typed table on self-hosted ClickHouse or ClickHouse Cloud. Choose replace when the old table must remain visible until the new snapshot is ready. Append and upsert write directly to the live table and can be visible before the run finishes.

​
Configuration

A database left empty is defaulted by the server to the normalized source connection name. The sink always authenticates against the default database and runs CREATE DATABASE IF NOT EXISTS for the destination from there, so the configured user needs access to default even when writing elsewhere. The sink prefers batches of 10,000 rows.

​
Supported write modes

CDC and incremental delete are not supported because ClickHouse does not offer efficient row-level deletes for this sink. See replication modes.

​
Table management

EnsureSchema behavior depends on the write mode. For replace, the run creates a staging table named __filament_stage_<hex8> (run-scoped) and leaves the destination untouched until Commit. For append and upsert, the live table is created if missing:
with tuple() as the sorting key for keyless resources. Upsert modes use ReplacingMergeTree() under the insert_order version strategy, or ReplacingMergeTree(<cursor field>) under the cursor strategy. The version field must exist in the schema, be NOT NULL, and be a timestamp or timestamptz. A pre-existing live table is validated, and mismatches are hard errors:
  • The engine must be MergeTree for append or ReplacingMergeTree for upsert (Shared* cloud variants are accepted).
  • A ReplacingMergeTree version argument must match the resolved version policy exactly.
  • The table’s sorting key must equal the sorted source primary key.
Missing columns are added to live tables with ADD COLUMN IF NOT EXISTS. Evolution is add-only.

​
Type mapping

Nullable fields are wrapped in Nullable().

​
Write mechanics and atomicity

The sink converts Arrow values to the destination’s typed columns. It accepts common textual forms for numbers, timestamps, and encoded bytes, but rejects a missing or null value for a non-nullable column. Append and upsert modes write directly to the live table, so partial data is visible mid-run. Replace mode is invisible until Commit, which per staged resource runs RENAME TABLE when the destination does not exist yet, or EXCHANGE TABLES followed by DROP TABLE of the stage. The exchange is atomic and requires the destination database to use the Atomic engine (the default).
ReplacingMergeTree deduplicates at background merge time, not on insert. Until parts merge, a key can have multiple rows — query with SELECT … FROM tbl FINAL to read the current state.

​
Failure behavior

Abort drops the run’s staging tables and nothing else. Failed replace runs leave the destination exactly as it was. Failed append or upsert runs can leave partial rows in live tables, which upsert re-runs converge through the primary key.