connectors/iceberg writes records as Parquet data files into Iceberg
tables, via any catalog backend registered with
iceberg-go.
Catalog backends
Configured through acatalog.provider field, which switches which config
fields are shown:
Write modes
write_mode controls how each commit applies a resource’s staged records:
auto (the default) resolves to replace for a full-mode run or append
otherwise, or it can be pinned explicitly to append, replace, upsert,
delete, or merge.
Staged, transactional commits
The sink implementsfilament.Transactional: Stage opens a scope, Write
buffers each resource’s records into it (spilling to disk past a configured
byte threshold), and Promote commits every resource in the stage as
Iceberg transactions — dropping the stage only once every resource
succeeds, so a retry after a partial failure skips whatever already
committed. Used directly (no explicit Stage call), Commit opens and
promotes an implicit stage itself.
Row buffers are parsed and streamed to the Iceberg writer in bounded
8192-row chunks (via Arrow record batches) rather than materialized as one
giant table in memory, so a large resource doesn’t require a matching
memory footprint.
Schema evolution
EnsureSchema is add-only: a source column not yet on the table gets
added, but type changes, nullability changes, and dropped columns are never
applied automatically. Iceberg tolerates a reader seeing a superset schema;
a narrowing change could silently break something reading the table, so it’s
left to a deliberate migration instead.
Modes
SupportsSnapshotReplace, Append, SnapshotUpsert, Upsert, Delete,
and CDC write policies (see Replication
modes).