Skip to main content
Connector ID mysql connectors/mysql/sink (registered as mysql, alpha, though the MySQL source is beta) loads each resource into its own typed MySQL table with native column types, primary keys, and NOT NULL constraints from the source schema. It requires MySQL 8.0 or newer.

​
Configuration

Filament must be able to resolve a destination database. If database is not set on the pipeline, it uses the database in the DSN or the normalized source connection name.
The MySQL server must have local_infile=ON. Stock community installations often disable it, while many managed services enable it. The sink cannot load data when this setting is off.

​
Loading

Before reading from the source, Filament creates or updates one typed table per resource. It then sends each Arrow batch through LOAD DATA LOCAL INFILE as tab-separated values held in memory; no temporary file or JSON conversion is involved. MySQL can turn some load problems into warnings. After every load, Filament compares the accepted and submitted row counts and checks the session warnings. It fails the batch if rows were skipped or values were coerced. Binary columns arrive as their raw bytes; JSON columns pass through unchanged so nested structure survives the round trip. The sink pins its session to UTC, matching the source, so a same-engine timestamp column round-trips as the same instant. Filament creates the destination database if needed. The sink prefers batches of 4,096 rows.

​
Supported write modes

The sink advertises full_replace, full_append, full_upsert, incremental_upsert, cdc_append, and cdc_merge. See replication modes. They bind to three behaviors:
  • full_replace replaces the destination; full_append inserts without removing existing rows.
  • full_upsert and incremental_upsert insert with LOAD DATA … REPLACE, making re-runs and retries idempotent.
  • cdc_append retains every change as a history row. cdc_merge maintains current state: inserts and updates upsert, and deletes delete.

​
Table management

With schema_evolution disabled, EnsureSchema runs per resource before extraction.
  1. CREATE TABLE IF NOT EXISTS with typed columns and a primary key.
  2. TRUNCATE — only for replace. Append preserves existing rows; upsert and merge preserve resumable state.
  3. Missing columns are added by checking information_schema.columns and issuing ALTER TABLE … ADD COLUMN for each absent one (MySQL has no ADD COLUMN IF NOT EXISTS). Evolution is add-only. Incompatible type changes are not applied and fail later if a write cannot cast the value.

​
Checked schema evolution

Set schema_evolution to strict or nullable_add to compare the complete source model with the destination before changing it. A new table can be created in either mode. Existing tables gain only nullable fields under nullable_add. Drops, type changes, key changes, and nullability changes block. The full diff must pass before any additions or replace-mode truncation. Checked mode supports ordinary InnoDB tables and compiler-owned scalar types. Text columns use utf8mb4_bin. Unsupported native declarations, column defaults, generated columns, auto-increment, other collations, partitions, triggers, and non-primary constraints block instead of being approximated. Each batch must match the prepared Arrow layout. The sink rechecks the destination under a metadata lock held through its write transaction. MySQL DDL commits separately from data writes. All approved additions use one ALTER TABLE, then the sink reacquires its table lock and verifies the result. If the run fails afterward, added columns can remain. A retry inspects the current table and only applies remaining additions. This does not provide PostgreSQL’s transactional DDL rollback. Replace mode still clears rows after successful schema validation and retains its existing abort behavior. The destination account needs LOCK TABLES in addition to its existing DDL and write permissions. Schema detections and blocks use the shared event catalog. This option applies to bounded runs. The MySQL sink does not yet implement continuous epochs, so it cannot receive the live NATS evolution path.

​
Type mapping

timestamptz maps to datetime(6) rather than MySQL’s timestamp because timestamp cannot represent values past 2038. Native types from other engines are ignored. Only MySQL-native declarations pass through.

​
Atomicity and integrity

Upserts use MySQL’s REPLACE behavior. JSON values are converted to utf8mb4 at the load boundary. The sink verifies the exact load payload before sending it and returns that encoded checksum with the Arrow batch checksum. This verifies serialization at the MySQL write boundary; it does not read the committed table back. CDC batches are split into runs of consecutive deletes and non-deletes, applied in arrival order in one transaction per batch. Deletes use a multi-table DELETE joined against a per-session temporary table containing the run’s keys. Non-CDC batches commit as they are applied, so partial data is visible mid-run. Delivery is at-least-once, and upsert and merge modes converge through the primary key.

​
Failure behavior

On a failed run, Abort truncates replace tables. Append tables retain rows already written, so retrying a non-checkpointed append as a new run can produce duplicates. Upsert and merge tables are preserved because retries converge through the primary key; resumable extraction failures skip Abort entirely.