Skip to main content
connectors/mysql/sink (registered as mysql, alpha, though the MySQL source is beta) loads each resource into its own typed MySQL table with native column types, primary keys, and NOT NULL constraints from the source schema. It requires MySQL 8.0 or newer.

​
Configuration

Filament must be able to resolve a destination database. If database is not set on the pipeline, it uses the database in the DSN or the normalized source connection name.
The MySQL server must have local_infile=ON. Stock community installations often disable it, while many managed services enable it. The sink cannot load data when this setting is off.

​
Loading

Before reading from the source, Filament creates or updates one typed table per resource. It then sends each Arrow batch through LOAD DATA LOCAL INFILE as tab-separated values held in memory; no temporary file or JSON conversion is involved. MySQL can turn some load problems into warnings. After every load, Filament compares the accepted and submitted row counts and checks the session warnings. It fails the batch if rows were skipped or values were coerced. Binary columns arrive as their raw bytes; JSON columns pass through unchanged so nested structure survives the round trip. The sink pins its session to UTC, matching the source, so a same-engine timestamp column round-trips as the same instant. Filament creates the destination database if needed. The sink prefers batches of 4,096 rows.

​
Supported write modes

The sink advertises full_replace, full_append, full_upsert, incremental_upsert, cdc_append, and cdc_merge. See replication modes. They bind to three behaviors:
  • full_replace replaces the destination; full_append inserts without removing existing rows.
  • full_upsert and incremental_upsert insert with LOAD DATA … REPLACE, making re-runs and retries idempotent.
  • cdc_append retains every change as a history row. cdc_merge maintains current state: inserts and updates upsert, and deletes delete.

​
Table management

Before extraction, EnsureSchema runs per resource:
  1. CREATE TABLE IF NOT EXISTS with typed columns and a primary key.
  2. TRUNCATE — only for replace. Append preserves existing rows; upsert and merge preserve resumable state.
  3. Missing columns are added by checking information_schema.columns and issuing ALTER TABLE … ADD COLUMN for each absent one (MySQL has no ADD COLUMN IF NOT EXISTS). Evolution is add-only. Incompatible type changes are not applied and fail later if a write cannot cast the value.

​
Type mapping

timestamptz maps to datetime(6) rather than MySQL’s timestamp because timestamp cannot represent values past 2038. Native types from other engines are ignored. Only MySQL-native declarations pass through.

​
Atomicity and integrity

Upserts use MySQL’s REPLACE behavior. JSON values are converted to utf8mb4 at the load boundary. The sink verifies the exact load payload before sending it and returns that encoded checksum with the Arrow batch checksum. This verifies serialization at the MySQL write boundary; it does not read the committed table back. CDC batches are split into runs of consecutive deletes and non-deletes, applied in arrival order in one transaction per batch. Deletes use a multi-table DELETE joined against a per-session temporary table containing the run’s keys. Non-CDC batches commit as they are applied, so partial data is visible mid-run. Delivery is at-least-once, and upsert and merge modes converge through the primary key.

​
Failure behavior

On a failed run, Abort truncates replace tables. Append tables retain rows already written, so retrying a non-checkpointed append as a new run can produce duplicates. Upsert and merge tables are preserved because retries converge through the primary key; resumable extraction failures skip Abort entirely.