Skip to main content
The MySQL source reads from MySQL 8.0+ with resumable keyset snapshots, timestamp-watermark incremental reads, and binlog CDC. The connector is beta.

​
Configuration

​
Snapshot reads

Tables with a primary key use ordered keyset reads. This works well with InnoDB because the primary key also determines physical row order. Filament saves the last key so an interrupted read can resume. Large tables can be divided into as many as 64 primary-key ranges. Integer keys use numeric boundaries; UUID, text, and datetime keys use sampled ranges with roughly equal row counts. Composite keys are also supported. Each range uses its own consistent, read-only snapshot. Parallel ranges may therefore observe the table at slightly different moments; the MySQL source does not provide one shared snapshot across the entire run. Binary values remain raw bytes when they are written into Arrow batches. A table without a primary key is read in a single streaming pass. That pass is not resumable, so a resume re-reads the whole table.

​
Incremental reads

Incremental resources require both a primary key and a non-null DATETIME or TIMESTAMP cursor column. Conventional names such as updated_at, modified_at, and last_modified_at are recommended automatically; a cursor can also be selected per pipeline resource. The column must advance on every insert and update that the pipeline needs to observe. The first incremental run captures the current (cursor, primary key...) high watermark, then performs a resumable primary-key backfill. After that backfill completes, scheduled runs page by the compound watermark, which keeps rows sharing the same timestamp deterministic. Each run bounds itself at a high watermark inside a consistent-snapshot transaction. The default lookback is zero, so incremental append does not duplicate an overlap window. Configure a positive per-resource lookback to catch late commits; when a lookback is enabled, incremental upsert is the recommended write policy because recent rows are deliberately re-read. For large tables, an index beginning with the cursor and followed by the primary-key columns avoids an incremental filesort, for example:

​
CDC

CDC connects as a binlog replica (go-mysql BinlogSyncer) and decodes insert, update, and delete events into the same columns the snapshot reader produces. Runs have catch-up semantics. Each captures the server’s current binlog position as a watermark, streams from the checkpoint up to it, and returns. Continuous CDC is a re-request loop, so it can run on a schedule. No-change runs still persist the watermark. An existing file-position checkpoint keeps that format if GTID is enabled later. GTID checkpoints advance after a committed transaction. A crash during a transaction replays the whole transaction, so use merge writes that converge by primary key. When a selected table has no CDC checkpoint, the run first reads it in full through a consistent snapshot, and the change stream then continues from the binlog position captured just before that snapshot opened. To make the two agree, the run takes read locks on those tables in a side session for the instant between capturing the position and opening the snapshot. Writers of those tables wait for a couple of round trips and nothing else is blocked. The baseline and the stream then share one exact point in both cursor modes, so a pipeline can start on CDC alone without a separate full run. Run row limits do not apply to this bootstrap read. The tables must use InnoDB, since only a transactional engine gives the snapshot one point in time. Binlog retention must cover the time the bootstrap read takes, or the stream cannot start from the captured position. snapshot_mode=none skips that read. A table without a checkpoint streams from the binlog position at the start of its first run, so the sink starts empty and fills in as rows change. Updates carry the full row, so a merge sink upserts rows it has never seen; deletes of unseen rows are no-ops. This mode needs neither LOCK TABLES nor InnoDB. Requirements:
  • ROW binlog format (the 8.0+ default)
  • Binlog retention covering the resume window
  • REPLICATION SLAVE and REPLICATION CLIENT privileges
  • LOCK TABLES and SELECT on the CDC tables, held for a moment when a table is first bootstrapped
  • A TCP connection (unix sockets are rejected for CDC)
Behavior worth knowing:
  • A primary-key-changing update is emitted as a delete of the old row plus an insert of the new one.
  • Column names are not in the binlog. Each table’s column list is read from information_schema (this works with binlog_row_metadata=MINIMAL) and the cache is invalidated on any DDL event.
  • A column-count mismatch between a binlog event and the cached schema fails the run rather than mis-mapping values. Re-snapshot the table after incompatible DDL.
See Replication modes for how modes map to write policies.