Skip to main content
The BigQuery sink (alpha) writes each resource to a typed BigQuery table. It supports bounded snapshot, incremental, and CDC runs using the BigQuery Storage Write API. Arrow batches stream into temporary tables, then one query per resource publishes the results at run commit.

​
BigQuery setup

Authentication uses Application Default Credentials, so the connection has no credential fields. On GKE, use Workload Identity. For local development, run gcloud auth application-default login before starting Filament. Grant the credential roles/bigquery.jobUser on the project and roles/bigquery.dataEditor on the project or on a pre-existing destination dataset. Creating a new dataset requires the project-level grant. Enable the BigQuery API and BigQuery Storage API (bigquerystorage.googleapis.com) in the project. Storage writes require bigquery.tables.updateData, included in the data editor role. TestConnection authenticates and executes SELECT 1.

​
Configuration

The sink prefers batches of 20,000 rows or approximately 8 MiB, whichever limit is reached first, with a ten-second flush interval. Run-level overrides take precedence. Remaining rows flush when extraction finishes or at source checkpoint boundaries.

​
Supported write modes

The sink advertises full_replace, full_append, full_upsert, incremental_append, incremental_upsert, incremental_delete, cdc_append, and cdc_merge. See replication modes.
  • Replace publishes the entire resource with a transactional truncate and insert.
  • Append publishes all of the resource’s staged rows with one insert query.
  • Upsert, delete, and merge publish with one MERGE, folding repeated keys across all batches to their last operation. These modes require a primary key.

​
Table management

The sink creates the destination dataset if needed. Resource workers create missing destination tables and add missing columns through metadata API calls. These operations do not submit DDL query jobs. New columns on existing tables are nullable; existing columns are not dropped, renamed, or altered. Incompatible existing types fail the run. Filament uses primary-key columns for merge conditions without adding BigQuery key constraints. Schemas remain limited to 16 key columns of key-eligible types. Each populated resource gets an isolated staging table with a seven-day expiration, normally deleted when publication completes.

​
Type mapping

​
Write mechanics and atomicity

Arrow IPC messages stream over reusable gRPC connections with gzip transport compression. Request size limits and the memory budget apply to the uncompressed messages; compression reduces network traffic without changing integrity checks. Each resource has an ordered stream with a sliding window of up to four outstanding append requests. Up to eight append requests can be in flight across the sink, including multiple requests for the same table. Table metadata and stream creation have a separate eight-resource limit so new table setup can overlap with established uploads. The sink bounds queued and in-flight encoded payloads to 64 MiB, plus active encoding and transport buffers. Per-resource queues share this byte budget without a separate batch-count cap, so a few small batches for a slow table do not block all other tables. Encoded batches split below an 8 MiB request budget; a single row that exceeds that budget fails with an explicit error. Uploads use pending streams with explicit row offsets. Transient append failures retry at the same offset, including when BigQuery accepted a request but its response was lost. A connection failure stops both sending and receiving before replaying the unacknowledged suffix. Already-acknowledged requests are retired, and their memory is released immediately. The four-request window refills after each acknowledgement; it does not wait for all four requests to finish. At commit, the sink waits for uploads, finalizes each stream, checks its row count, and commits it to the staging table. Up to eight independent tables then publish concurrently, using one query job per table. Empty append or upsert resources require no query; an empty replacement truncates its destination. Destination rows become visible at publication. Write receipts and batch progress reported during extraction describe encoded, queued batches; they do not mean the rows are already queryable in the destination. All policies defer durable checkpoints until the entire run commits successfully. Receipts include an Arrow IPC checksum; the sink does not read destination rows back for verification. Publication is atomic per table, not across the dataset. No external storage bucket is required. TIME and decimal values use lossless strings in staging and cast to their destination types during publication.

​
Failure behavior

A failure before publication leaves destination rows unchanged. A failure during publication can leave some tables updated while others retain their previous contents. Additive schema changes can remain after either kind of failure. Publication job IDs include the run, dispatch execution, destination, and write mode. Resuming a paused run creates a new execution and publishes a new segment. Redelivery of the same execution can reuse its job only when a persisted SHA-256 fingerprint matches the staged rows, schema, operations, and batch checkpoints. Changed replay contents or batch boundaries fail instead of acknowledging rows that the earlier job did not publish. A new run ID is a new operation: replaying an append run can produce duplicates, while keyed writes converge through their keys. Failed runs replay extraction because their pending-stream progress is not durably resumable. Pausing extraction leaves accepted uploads running so Commit can drain them. Commit cancellation and Abort cancel uploads. Abort removes staging tables that are safe to delete. If a publication might still be running, its staging table remains available for job recovery until expiration. A recorded failed publication job is surfaced as a failure on retries of that execution; a corrected attempt needs a new execution ID. Pending streams must be committed within 24 hours. This path is intended for bounded runs, not indefinite continuous ingestion. See Google’s pending-stream lifecycle for buffer limits and transaction behavior.

​
Troubleshooting

  • SERVICE_DISABLED for the Storage API means bigquerystorage.googleapis.com must be enabled in the project.
  • could not find default credentials means Application Default Credentials are missing from Filament’s environment. Attach a service account, configure Workload Identity, or run gcloud auth application-default login.
  • Permission errors on jobs or datasets usually mean the credential lacks roles/bigquery.jobUser on the project or roles/bigquery.dataEditor on the destination dataset.
  • A dataset-not-found error naming an unexpected location means the dataset lives somewhere other than the connection’s location value. Match the connection to the dataset or leave location empty.
  • Dataset IDs allow only letters, numbers, and underscores, so hyphens or dots in the pipeline’s dataset value are rejected.

​
Diagnosing slow writes

bigquery.write.ready confirms the worker is using the Storage Write API path and reports compression, the global request limit, and inflight_per_stream. bigquery.write.prepare.started and bigquery.write.publish.started identify resource setup and final publication. Operations lasting at least one second emit bigquery.write.timing at INFO with their resource, phase, and duration. Queue timings include backpressure_ms; append timings include rows, uncompressed bytes, send_ms, and response_wait_ms. Sending includes compression and handing data to gRPC; response wait includes remaining network transfer and service processing, so it is not a measurement of BigQuery processing alone. DEBUG includes fast operations too. bigquery.write.retry reports failed append attempts and whether they can be retried. These events contain no row payloads.