> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Iceberg

> Write Parquet data files to Apache Iceberg tables

The Iceberg sink (**alpha**) writes Arrow batches as Parquet data files through
a REST or [iceberg-go](https://github.com/apache/iceberg-go)-registered
catalog. Rows remain staged until their resource commits.

<Warning>
  Atomicity is per resource, not per pipeline run. If a run moves several
  resources, an earlier table can commit before a later table fails. Namespace,
  table, and add-only schema changes can also become visible before data is
  committed. Filament does not provide one exactly-once transaction across the
  route.
</Warning>

## Configuration

| Field | Scope | Default | Description |
| - | - | - | - |
| `catalog` | Connection | — | Required. Catalog connection object. See below. |
| `catalog.provider` | Connection | — | Required. `generic`, `rest`, `polaris`, or `lakekeeper`. |
| `table.location_root` | Connection | — | Optional object-storage root for new tables (e.g. `s3://bucket/warehouse`). Leave empty when the catalog assigns locations. |
| `namespace` | Pipeline | `default` | Destination namespace, dot-separated |
| `stage_buffer_limit_mb` | Pipeline | `256` | In-memory buffer threshold in MiB before spilling to temp files |

A `namespace` left empty is defaulted server-side to the normalized source
connection name.

Per provider, inside `catalog`:

| Provider | Fields |
| - | - |
| `generic` | `type` or `uri` (either is required). Optional `warehouse`, `properties`. |
| `rest` | `uri` required. Optional `warehouse`, `auth`, `properties`. |
| `polaris` | `uri` and `warehouse` required, plus `auth` |
| `lakekeeper` | `uri` and `warehouse` required, plus `auth` |

For `generic`, `type` is an iceberg-go catalog type (`sql`, `hive`, `glue`, …)
and `properties` is free-form, passed to iceberg-go. For `polaris`, `uri` is
the REST endpoint (typically ending `/api/catalog`) and `warehouse` is the
catalog name in Polaris. For `lakekeeper`, `uri` ends in `/catalog`.

REST-style providers share an `auth` object:

| Field | Default | Description |
| - | - | - |
| `auth.type` | `none` | `none`, `bearer`, or `client_credentials` |
| `auth.token` | — | Bearer token. Required for `bearer`. Secret. |
| `auth.client_id` / `auth.client_secret` | — | Required for `client_credentials`. `client_secret` is secret. |
| `auth.token_uri` | — | OAuth token endpoint. Empty uses the catalog's default. |
| `auth.scope` | — | Optional OAuth scope (Polaris commonly uses `PRINCIPAL_ROLE:ALL`) |
| `auth.audience` / `auth.resource` | — | Optional OAuth audience and resource indicators |

Data files go to the table's storage via the catalog's FileIO. The `s3`/`s3a`/
`s3n`, `gs`, and `azure` schemes are supported. `TestConnection` loads the
catalog and lists namespaces.

## Supported write modes

The sink supports full replace, append, and upsert; incremental upsert and
delete; and CDC append and merge. Incremental delete can hard-delete rows from
Iceberg tables. See
[replication modes](/pages/guides/concepts/replication-modes).

## Table management

`EnsureSchema` creates the namespace and the table with
`write.format.default=parquet`. For an existing table, evolution is
**add-only**. New fields are appended to the schema, while type changes,
nullability changes, and drops are never applied. A resource whose name
collides with Iceberg's reserved metadata names gets a `_` suffix.

### Type mapping

| Logical type | Iceberg type |
| - | - |
| `bool` | `boolean` |
| `int16` / `int32` | `int` |
| `int64` | `long` |
| `float32` / `float64` | `float` / `double` |
| `decimal` | `decimal(p,s)` (default 38,9) |
| `string` / `json` | `string` |
| `bytes` | `binary` |
| `date` / `time` / `timestamp` / `timestamptz` / `uuid` | corresponding primitives |
| `array` | `string` (source-native literal text) |
| unknown | `string` |

Primary key fields become the table's identifier fields.

Array columns are written as their source's literal text. A table created by an older
Filament version with a `list` column for the same source column must be
recreated; commits fail on the type mismatch rather than rewriting the column.

## Write mechanics and atomicity

The sink buffers each resource's Arrow batches and spills them to a compressed
temporary file after `stage_buffer_limit_mb`. At commit, it writes the buffered
rows to Parquet and updates the Iceberg table.

`EnsureSchema` is outside the stage. It may create the namespace or table, or
commit add-only schema evolution, before extraction begins. Those catalog
changes can therefore be visible even if the later run fails.

Buffering validates operations against the bound mode:

* append rejects updates and deletes
* replace rejects deletes
* upsert, delete, and merge require key fields

At commit, the sink converts batches to the table's Arrow schema and streams
them to the Parquet writer. It reloads each table from the catalog first so it
uses current storage credentials, then opens one Iceberg transaction:

* **append** adds Arrow record batches in 8,192-row chunks.
* **replace** overwrites the table with the run's records. A replace run that
  produced **zero records still overwrites the table to empty**. An empty
  source empties the destination.
* **upsert, delete, and merge** keep the latest operation for each key, then
  update or delete matching rows.

Buffered rows are invisible until their resource transaction commits. A run
with several resources commits them sequentially, so a later resource can fail
after an earlier one is already visible. Repeating `Commit` on the same open
sink skips resources it already committed, but the engine does not provide one
atomic transaction or exactly-once boundary across the route.

## Failure behavior

`Abort` discards in-memory buffers and temporary spill files. It does not undo
namespace/table creation, schema evolution from `EnsureSchema`, or any resource
transaction that committed before a later resource failed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.