Meilisearch

The Meilisearch sink (alpha) writes each resource to a Meilisearch index, one document per row. Documents are sent as NDJSON and indexed by Meilisearch’s asynchronous task queue. By default the run waits for those tasks at commit, so a successful run means the documents are searchable.

​

Configuration

An index_prefix left empty is defaulted by the server to the normalized source connection name. api_key can be omitted for a development instance that runs without a master key. TestConnection calls /health. That endpoint does not require authentication, so a passing test proves the server is reachable but not that the key can create indexes or write documents.

​

Supported write modes

Every mode writes through Meilisearch’s add or replace documents operation. A document whose primary key already exists is replaced in full, and a new key is added. See replication modes.

​

Index management

EnsureSchema resolves one index per resource and creates it when missing. Existing indexes keep their settings. For full_replace, an existing index has all of its documents deleted once per run, before the run’s first write. The index UID is index when set. Otherwise it is index_prefix and the resource name joined with _. Characters outside letters, digits, -, and _ become _, and leading or trailing separators are trimmed. With prefix local_pg, the resource public.accounts writes to local_pg_public_accounts. Meilisearch is schemaless, so there is no column evolution. New source fields appear as new document attributes. The sink does not manage searchable, filterable, or sortable attributes, ranking rules, or any other index setting. Configure those in Meilisearch.

​

Primary key

Meilisearch requires one primary key attribute per index. The sink resolves it in this order:

1. The primary key of an index that already exists
2. primary_key from the pipeline configuration
3. The source primary key, when it is a single field
4. The first field named id, ignoring case
5. The first field ending in _id or Id
6. The first field in the schema

Composite source keys are not combined. A resource with a composite key falls through to the later rules, so set primary_key for it explicitly. Primary key values must satisfy Meilisearch’s document id format of letters, digits, -, and _.

​

Value encoding

Documents carry source field names and JSON values.

​

Write mechanics and atomicity

Each batch is encoded as NDJSON and sent in one request, gzip-compressed by default. Meilisearch accepts the request, returns a task, and indexes the documents in the background in queue order. For cdc_merge, a batch is sent as alternating delete and document requests that follow the order of its operations, so a key that is deleted and inserted again in one batch ends up present. A merge on a resource whose primary key is not in the schema is an error. At Commit the sink waits for every task from the run to succeed, up to task_timeout. A failed or canceled task fails the run with Meilisearch’s error code and message. Malformed documents and invalid primary key values are reported here, not when the batch is sent, because Meilisearch validates them during indexing. With wait_for_tasks set to false, the run succeeds as soon as every request is accepted. Indexing continues after the run ends, and indexing errors are visible only in the Meilisearch task list. Documents become searchable as their tasks finish, so partial data is visible mid-run. There is no staging index and no swap. During a full_replace run the index is empty, then partial, until the run’s tasks finish.

​

Failure behavior

Abort stops waiting on tasks and nothing else. Tasks already accepted by Meilisearch keep running, and documents already indexed stay in the index. A rerun converges through the primary key, because every write replaces the document with the same key. A failed full_replace run leaves the index cleared or partly loaded until the next successful run. A task_timeout expiry fails the run but does not cancel the tasks. Meilisearch may still finish them. Raise the timeout for large first loads, where indexing takes much longer than sending.