Skip to main content
A sink owns the destination boundary. Filament decides when a batch is ready, which write policy applies, and whether the run should commit or abort. The sink turns the borrowed Arrow batch into a durable destination write and reports what it handled.

​
The run lifecycle

Every sink implements:
The normal sequence is:
  1. Open prepares one run.
  2. A schema-aware sink receives EnsureSchema once per selected resource.
  3. Apply handles each batch under its resource’s write policy.
  4. Commit establishes the sink’s success boundary.
  5. Abort cleans up a run that cannot be safely continued.
Some recovery-safe failures preserve accepted writes instead of calling Abort. That can happen only when every resource can resume and replay will not duplicate append-only data or expose uncommitted work. A sink should still make Abort safe because it also runs after setup and commit errors.

​
Applying a batch

arrowbatch.Batch contains one resource and part, an Arrow record batch, an operation vector, sequence metadata, and an optional checkpoint delta. A simple append sink can look like this:
The example assumes writeRows computes and verifies encodedCRC over the serialized bytes immediately before transport. A sink that does not advertise encoded integrity omits EncodedCRC.

​
Borrowing and ownership

Apply borrows the batch for the duration of the call. Its Arrow rows and operation vector are valid until Apply returns. If a sink buffers a batch for Commit, it must call Retain before returning and eventually call Release. This rule matters for sinks such as Iceberg that hold batches until commit. It is also why Batch is passed by pointer.

​
Integrity evidence

Two checks describe different boundaries:
  • WriteCRC is required. Recompute batch.IntegrityCRC() at the sink’s final in-memory boundary. The pipeline compares it with the checksum calculated before calling Apply.
  • EncodedCRC is optional unless SinkCapabilities.EncodedIntegrity is true. It covers the exact serialized bytes handed to the transport.
Neither value is a universal read-after-write verification of the destination. A sink that needs that stronger guarantee must implement destination-specific acknowledgement or validation as part of its write.

​
Honor the write policy

ApplyOptions.Policy tells the sink how to write this resource:
  • append retains every accepted row.
  • replace produces a fresh resource snapshot.
  • upsert keeps a current row per primary key.
  • merge applies ordered CDC inserts, updates, and deletes.
  • delete removes rows by key for the internal incremental-delete path.
Call ValidateBatch before writing. Policy.Keys contains resolved primary-key columns when the mode needs them, and Policy.Version explains how an upsert resolves competing values. CDC can bind either append or merge. Append is change history; merge is current state.

​
Batching and concurrency

The engine flushes a builder when it reaches the run’s row or byte limit. If neither is configured, it uses the sink’s preferred row count and then the engine default. The periodic flush request is observed at the next completed row; a source with an idle stream calls Flush itself. When snapshot parallelism is greater than one, Apply can run concurrently. Protect shared buffers and accounting, and size connection pools in Open. Ordered CDC, incremental, and checkpointed runs are forced to one writer.

​
Capabilities and registration

SinkSpec.Capabilities.WritePolicies advertises the modes the sink really implements. Capabilities can also declare whether writes survive after each Apply or only after Commit; that durability boundary determines when a checkpoint may advance. Register the sink through a fresh-instance factory:
The connectors/stdout package is the smallest complete sink. It demonstrates Arrow-to-NDJSON serialization, both integrity values, concurrency protection, and the complete run lifecycle.