> ## Documentation Index
> Fetch the complete documentation index at: https://filament.getgalaxy.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Runs and recovery

> Run states, recovery boundaries, and lifecycle controls

A run is one execution of one pipeline route. The useful question after an
interruption is not simply “was there a checkpoint?” It is “can every resource
in this route continue without losing or duplicating accepted work?”

## Run states

| State       | What it means                                          | What can happen next                             |
| ----------- | ------------------------------------------------------ | ------------------------------------------------ |
| `scheduled` | Created for a future schedule                          | Scheduler requests it                            |
| `requested` | Persisted and waiting for dispatch                     | Worker starts, or the run is paused/canceled     |
| `running`   | A worker owns the execution                            | Complete, fail, become partial, pause, or cancel |
| `partial`   | Extraction failed but all progress is safely resumable | Resume the same run                              |
| `paused`    | Work stopped intentionally                             | Resume or cancel                                 |
| `completed` | Sink committed successfully                            | Terminal, except CDC catch-up can run again      |
| `failed`    | Work cannot safely continue as the same attempt        | Terminal                                         |
| `canceled`  | Work was intentionally aborted                         | Terminal                                         |

An idempotency key derives a stable run ID for a tenant. Repeating the same
request does not create another logical attempt once that run has progressed.

## What one attempt does

The runner resolves secrets and connectors, plans resources and ingestion
behavior, ensures destination schemas, seeds checkpoint state, and starts the
Arrow pipeline. It calls `Commit` only after extraction and writes succeed. A
non-resumable failure ends with `Abort`. Filament gives cleanup its own short
timeout, even when the original run context has already been canceled.

Schema preparation is intentionally before extraction. Depending on the sink,
`EnsureSchema` may create or evolve destination objects before the final commit;
connector docs should state their own visibility and cleanup behavior.

## When failure becomes partial

A failed standard run becomes `partial` only when all resources have progress
that is safe to preserve:

* every resource has checkpoint coverage;
* none waits until `Commit` to make its checkpoint durable;
* none uses append, where replay could duplicate rows; and
* at least one resource is incremental or otherwise checkpointed.

This route-wide rule matters for mixed plans. One resumable `users` resource
does not make a neighboring full-replace `orders` resource resumable. When the
rule is not satisfied, the run fails and a new attempt starts those resources
from their normal beginning.

CDC saves its stream position only after a successful commit, so a failed
attempt is not labeled partial. A later CDC catch-up starts from the last
committed position. Even a successful cycle with no changes can advance that
position.

## Pause, resume, and cancel

`SignalRun` supports all three controls:

* Pausing a requested run changes its state immediately. Pausing a running run
  sends a worker command and waits for the worker's acknowledgement.
* A fully checkpointed running route drains accepted rows, commits them, and
  pauses with progress preserved. A checkpoint-free route aborts and later
  restarts. A route with mixed checkpoint coverage rejects pause because neither
  outcome is safe for every resource.
* Resuming a paused or partial run returns it to requested. Only checkpointed
  progress is retained; restart-only work has its execution totals reset.
* Canceling a running run stops extraction and aborts the sink before the worker
  acknowledges cancellation. Requested, paused, and partial runs can be
  canceled directly.

Signals are state-aware and idempotent where repeating the same intent is safe.
Invalid transitions, such as resuming a completed non-CDC run, are rejected.

## Failure containment

A source panic is converted to a run error. When a sink error causes extraction
to stop, the sink error remains the root cause. Terminal facts are published on
a short detached context so cancellation does not leave a run appearing to be
active indefinitely.

Under Kubernetes dispatch, workers emit heartbeats and the reaper fails stale
running jobs. Partial runs are exempt because they are intentionally waiting for
a resume request. In-process dispatch does not use the worker heartbeat/reaper
path.

## Progress events

The event stream includes run lifecycle and control facts, resource progress,
batch writes and integrity evidence, cursor/checkpoint changes, and pressure or
retry events. Examples include `run.pause_requested`, `run.paused`,
`batch.integrity_verified`, `batch.encoded_integrity_verified`, and
`cursor.checkpoint_saved`.

These events record worker execution and let the tracker update totals and progress.
The API also persists intake and control transitions directly, as described in
[Architecture](/pages/guides/concepts/architecture).