Skip to main content
A run is one execution of one pipeline route. The useful question after an interruption is not simply “was there a checkpoint?” It is “can every resource in this route continue without losing or duplicating accepted work?”

​
Run states

An idempotency key derives a stable run ID for a tenant. Repeating the same request does not create another logical attempt once that run has progressed.

​
What one attempt does

The runner resolves secrets and connectors, plans resources and ingestion behavior, ensures destination schemas, seeds checkpoint state, and starts the Arrow pipeline. It calls Commit only after extraction and writes succeed. A non-resumable failure ends with Abort. Filament gives cleanup its own short timeout, even when the original run context has already been canceled. Schema preparation is intentionally before extraction. Depending on the sink, EnsureSchema may create or evolve destination objects before the final commit; connector docs should state their own visibility and cleanup behavior.

​
When failure becomes partial

A failed standard run becomes partial only when all resources have progress that is safe to preserve:
  • every resource has checkpoint coverage;
  • none waits until Commit to make its checkpoint durable;
  • none uses append, where replay could duplicate rows; and
  • at least one resource is incremental or otherwise checkpointed.
This route-wide rule matters for mixed plans. One resumable users resource does not make a neighboring full-replace orders resource resumable. When the rule is not satisfied, the run fails and a new attempt starts those resources from their normal beginning. CDC saves its stream position only after a successful commit, so a failed attempt is not labeled partial. A later CDC catch-up starts from the last committed position. Even a successful cycle with no changes can advance that position.

​
Pause, resume, and cancel

SignalRun supports all three controls:
  • Pausing a requested run changes its state immediately. Pausing a running run sends a worker command and waits for the worker’s acknowledgement.
  • A fully checkpointed running route drains accepted rows, commits them, and pauses with progress preserved. A checkpoint-free route aborts and later restarts. A route with mixed checkpoint coverage rejects pause because neither outcome is safe for every resource.
  • Resuming a paused or partial run returns it to requested. Only checkpointed progress is retained; restart-only work has its execution totals reset.
  • Canceling a running run stops extraction and aborts the sink before the worker acknowledges cancellation. Requested, paused, and partial runs can be canceled directly.
Signals are state-aware and idempotent where repeating the same intent is safe. Invalid transitions, such as resuming a completed non-CDC run, are rejected.

​
Failure containment

A source panic is converted to a run error. When a sink error causes extraction to stop, the sink error remains the root cause. Terminal facts are published on a short detached context so cancellation does not leave a run appearing to be active indefinitely. Under Kubernetes dispatch, workers emit heartbeats and the reaper fails stale running jobs. Partial runs are exempt because they are intentionally waiting for a resume request. In-process dispatch does not use the worker heartbeat/reaper path.

​
Progress events

The event stream includes run lifecycle and control facts, resource progress, batch writes and integrity evidence, cursor/checkpoint changes, and pressure or retry events. Examples include run.pause_requested, run.paused, batch.integrity_verified, batch.encoded_integrity_verified, and cursor.checkpoint_saved. These events record worker execution and let the tracker update totals and progress. The API also persists intake and control transitions directly, as described in Architecture.