Skip to main content
Connector ID nats The nats source supports continuous execution. Select subject resources such as orders.> to let Filament manage durable filtered consumers. Configure the NATS url and authentication using the shared connection settings. Bounded extraction is unsupported.

Payload columns

JSON object members become source columns alongside the subject and Filament event metadata. By default, the first message fixes each resource’s schema. Missing fields become null. Incompatible payload changes fail before certification. Non-object payloads remain raw bytes in a source-owned payload column.

Live nullable fields

With a PostgreSQL sink, set its schema_evolution option to nullable_add to accept newly observed payload fields during continuous execution. Filament buffers and probes every message in an epoch before encoding any rows. The buffer is bounded by the epoch limits and the consumer’s pending-message credit. Heartbeats keep buffered messages alive while schema checks or migrations are waiting. An initial message such as {"id": 1} can therefore be followed by {"id": 2, "email": "a@example.com"}. The sink adds a nullable email column. Earlier rows and later messages without email have null in that column. Absence never removes an accepted field or makes it required. Strings and booleans use their scalar types. Numbers, objects, arrays, and null-only fields use JSON to preserve their values without guessing a narrower type. JSON fields can later contain other JSON values. An incompatible change to an accepted scalar field blocks the entire epoch before row emission. Raw-byte payloads cannot automatically switch to object columns or vice versa. Use strict to establish the layout from the initial whole batch and block later additions. Leave schema_evolution disabled to retain first-message inference. Resources spanning several physical streams share one accepted layout. The accepted payload layout is stored in the epoch certificate alongside source progress. A restarted worker restores it even when the first replayed message omits previously seen fields. The source acknowledges only after both destination commit and checkpoint persistence succeed. Destination and checkpoint commits remain separate, so existing duplicate-delivery and uncertain-commit recovery limitations still apply. Streams with earlier checkpoints that contain no accepted layout must start a new generation before enabling live evolution. Resuming saved layouts requires the same pipeline version and enabled schema negotiation. A schema block keeps the last certified progress unchanged and suppresses automatic retries. Resolve the issue before requesting another attempt.

Retention and recovery

Managed filtered consumers read retained matching messages. A stream can also hold unrelated subjects, so gaps in its sequence numbers are expected. Filament does not treat stream-wide deleted IDs or a moved first sequence as proof that a matching message was lost. JetStream’s stream-wide retention metadata cannot establish which subject a deleted message belonged to. Matching messages removed before consumption or recovery can therefore be lost without detection. Size retention for the longest processing lag and recovery outage you need to support. Certification and delayed acknowledgements protect delivered input; they cannot recover input the server has already removed. Recovery can replay input, so destinations must tolerate duplicates. Explicit stream/consumer bindings use an unfiltered durable pull consumer with explicit acknowledgement, deliver-all, unlimited delivery attempts, MaxAckPending=1, and AckWait of at least one second. These consumers retain strict stream-sequence gap detection and stop when continuity cannot be proven.

Epoch batching

With a reusable sink, workers default to 1,000 records and a one-second read budget. Sinks that reopen per boundary default to 100,000 records and 30 seconds. These are soft targets. Consumer credit can produce smaller batches. Embedded callers can override these values through RunOptions.EpochBoundary. Provide positive MaxRecords and MaxWait. MaxBytes and MaxAge can further limit NATS reads. These settings are persisted in the admitted run specification. Managed consumers allow up to 1,000 pending messages and older single-credit managed consumers are upgraded in place. A consumer’s credit caps each batch; explicit user-owned consumers keep their single-message credit. Multiple physical streams are serviced in turn, with one consumer contributing to each epoch. All delivered messages in a batch receive heartbeat extensions and remain unacknowledged until the destination commit and Filament’s epoch certification succeed. Failed or canceled reads do not acknowledge their tentative batch.