Iceberg

The Iceberg sink (alpha) writes Arrow batches as Parquet data files through a REST or iceberg-go-registered catalog. Rows remain staged until their resource commits.

​

Configuration

A namespace left empty is defaulted server-side to the normalized source connection name. Per provider, inside catalog: For generic, type is an iceberg-go catalog type (sql, hive, glue, …) and properties is free-form, passed to iceberg-go. For polaris, uri is the REST endpoint (typically ending /api/catalog) and warehouse is the catalog name in Polaris. For lakekeeper, uri ends in /catalog. REST-style providers share an auth object: Data files go to the table’s storage via the catalog’s FileIO. The s3/s3a/ s3n, gs, and azure schemes are supported. TestConnection loads the catalog and lists namespaces.

​

Supported write modes

The sink supports full replace, append, and upsert; incremental upsert and delete; and CDC append and merge. Incremental delete can hard-delete rows from Iceberg tables. See replication modes.

​

Table management

EnsureSchema creates the namespace and the table with write.format.default=parquet. For an existing table, evolution is add-only. New fields are appended to the schema, while type changes, nullability changes, and drops are never applied. A resource whose name collides with Iceberg’s reserved metadata names gets a _ suffix.

​

Type mapping

Primary key fields become the table’s identifier fields. Array columns are written as their source’s literal text. A table created by an older Filament version with a list column for the same source column must be recreated; commits fail on the type mismatch rather than rewriting the column.

​

Write mechanics and atomicity

The sink buffers each resource’s Arrow batches and spills them to a compressed temporary file after stage_buffer_limit_mb. At commit, it writes the buffered rows to Parquet and updates the Iceberg table. EnsureSchema is outside the stage. It may create the namespace or table, or commit add-only schema evolution, before extraction begins. Those catalog changes can therefore be visible even if the later run fails. Buffering validates operations against the bound mode:

• append rejects updates and deletes
• replace rejects deletes
• upsert, delete, and merge require key fields

At commit, the sink converts batches to the table’s Arrow schema and streams them to the Parquet writer. It reloads each table from the catalog first so it uses current storage credentials, then opens one Iceberg transaction:

• append adds Arrow record batches in 8,192-row chunks.
• replace overwrites the table with the run’s records. A replace run that produced zero records still overwrites the table to empty. An empty source empties the destination.
• upsert, delete, and merge keep the latest operation for each key, then update or delete matching rows.

Buffered rows are invisible until their resource transaction commits. A run with several resources commits them sequentially, so a later resource can fail after an earlier one is already visible. Repeating Commit on the same open sink skips resources it already committed, but the engine does not provide one atomic transaction or exactly-once boundary across the route.

​

Failure behavior

Abort discards in-memory buffers and temporary spill files. It does not undo namespace/table creation, schema evolution from EnsureSchema, or any resource transaction that committed before a later resource failed.