pipeline/pipeline_bench_test.go.
BenchmarkNullSinkPipeline
The top rung of the subtraction ladder:
20,000 ~80-byte JSON records, built once during setup so fixture cost isn’t
measured, pushed through a pipeline configured with BatchMaxRows: 1000
into a nullSink. FlushInterval is set to an hour so row-count and
Close drive batching rather than the timer — the benchmark measures
throughput, not flush timing.
records/sec (computed from b.Elapsed()) alongside standard
allocs/op.
BenchmarkNullSinkPipelineParallel
The same idea with concurrency: 40,000 records spread across 8 resources
(so the batcher’s sharding engages), pushed from a pool of goroutines
sized to match, at parallelism levels 1, 4, and 8 (b.Run sub-benchmarks).
This isolates how the pipeline itself scales — independent of the
shared-database contention a real end-to-end benchmark against an actual
sink would also be measuring.
Why a null sink
nullSink still does real integrity work — it recomputes the write-side
CRC exactly as a production sink would via filament.CRC32C — it just
discards the batch afterward instead of writing it anywhere, so memory
stays flat across b.N iterations. That keeps the benchmark honest about
the pipeline’s own cost (batching, the read-side CRC, integrity
verification) without a real destination’s I/O in the number.