+
+
+ benchkit — Reference
+ Documentation for the benchkit benchmarking toolkit: run harness, statistics,
+ time-series events, rate ramping, the result schema, and the sweep / plot
+ tooling that consumes it.
+ Documentation · gorums repository · doc/benchkit.html
+
+
+
+
benchkit is the benchmarking toolkit for gorums.
+ It provides the measurement control plane (Control service), the run harness, the
+ statistics library, the result schema, and the time-series event stream. benchmark/
+ defines the concrete gorums benchmarks on top of it; cmd/benchmark is the node binary;
+ cmd/sweep orchestrates multi-configuration runs across a cluster and, with
+ -plot, turns the results into CSV tables and a Typst report.
+
Module layout. All of the above lives in benchkit/, a second Go module in
+ the gorums repository with module path github.com/relab/gorums/benchkit. It imports
+ gorums; gorums never imports it, which keeps the root go.mod free of benchmarking and
+ SSH orchestration dependencies. Paths on this page are relative to that module root, so run
+ make targets from the repository root and sweep itself from
+ benchkit/.
+
+
+
1. Package map
+
+
+
+
+ | Location |
+ Responsibility |
+
+
+
+
+ . (package benchkit) |
+ Control service, run harness, statistics, sample stores, ticker, event buffer, pacer,
+ clock-sync, result schema, report I/O, time-series CSV rendering. |
+
+
+ proto/ |
+ The .proto sources: proto/benchkit/{benchkit,control}.proto and
+ proto/benchmark/benchmark.proto. Generated with -I proto, so the
+ import paths protoc records stay benchkit/benchkit.proto and
+ benchmark/benchmark.proto, and generated code is written back beside the
+ packages that own it. |
+
+
+ benchmark/ |
+ The concrete gorums benchmarks (QuorumCall, AsyncQuorumCall, SlowServer, Multicast, and the
+ symmetric peer-to-peer variants) plus their server and symmetric-topology setup. |
+
+
+ cmd/benchmark/ |
+ Node binary. Registers the standard flags, builds the targets, runs the selected benchmarks, and
+ writes a result file. Built with make benchmark. |
+
+
+ cmd/sweep/ |
+ Cartesian sweep orchestrator: builds/deploys the binary over SSH, launches nodes, collects
+ result files, prints an aggregated summary table, and — with -plot — generates
+ the Typst report (see §14). Imports benchkit for the result
+ schema, the read-time trim, the latency statistics, and the time-series renderer. Built with
+ make sweep. |
+
+
+ cmd/sweep/typst/gorumsplot.typ |
+ The cetz-plot helper library, embedded in the sweep binary and copied into each
+ report directory beside the generated report.typ. See §14. |
+
+
+
+
+
+
+
2. Design rationale
+
Why benchkit is shaped the way it is.
+
+
sweep touches a benchmark binary through exactly two protocol-neutral contracts: the
+ CLI flag contract it launches a node with (§9) and the
+ result-file format it reads back (§12). It launches processes
+ over SSH, waits for them to exit, downloads result files, and prints an aggregated table —
+ indifferent to whether the workload is a gorums quorum call, a PBFT commit, or a Paxos round.
+ Stabilizing those two contracts is what lets sweep drive any protocol without importing its code.
+
+
Behind the binary, the measurement control plane is split from the workload. The Control
+ service (Start / Stop / ClockSync) is workload-agnostic and lives
+ in the neutral benchkit package alongside the statistics library, the result schema, the run
+ harness, the pacer, and clock-offset correction. A protocol author writes only their workload
+ .proto and a workload callback for harness.Run, and reuses everything else
+ unchanged. benchmark/ is itself just the gorums consumer of that neutral surface.
+
+
+
+
Reusable across protocols
+
+ - The
Control service + server
+ Stats, offset correction, aggregation- The result schema (canonical)
+ - The run harness (flags + lifecycle + output)
+ - The pacer / rate limiting and the event stream
+
+
+
+
Protocol-specific
+
+ - The workload
.proto & RPCs
+ - The workload callback registered with the harness
+ - The consensus / replication machinery itself
+
+
+
+
+
Several observability capabilities are modelled on the relab/hotstuff harness — the
+ per-interval time series, the typed event stream, bounded-memory (hdr) statistics, and in-run rate
+ ramping (§5, §4, §7). The
+ out-of-band "live controller" orchestration model was considered and deliberately not adopted: the
+ fire-and-collect model keeps sweep language-neutral and simple, which is the property that lets it drive
+ foreign binaries at all.
+
+
+
+
3. Run architecture
+
+
A benchmark binary runs one continuous measurement from t=0 to t=end and
+ records everything it observes. A single run produces two data products from the same
+ Stats.AddLatency call, so workload handlers need no special wiring:
+
+
+
+
Aggregate store
+
+ - Covers the entire run; every sample is retained in
StatsMode_EXACT (the
+ default).
+ - Backing store is selectable:
exact samples or a bounded-memory hdr
+ histogram (§4).
+ - Becomes the flat measured fields of
Result (throughput, latencies, and the
+ statistics derived from them).
+
+
+
+
Interval event stream
+
+ - Also covers the whole run, sampled at the ticker interval (§6).
+ - Each tick emits a
ThroughputInterval and a LatencyInterval
+ (per-interval Welford, O(1)), time-stamped by offset.
+ - Events accumulate in memory and are attached to the per-benchmark
Result as a
+ repeated events field — no separate file.
+
+
+
+
+
harness timeline (single continuous run)
+ ─────────────────────────────────────────────────────────────────────────
+ t=0 t=end
+ │ │
+ ├───────────────────── continuous measurement ──────────────────────────┤
+ │ │
+ │ aggregate store : ▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ │
+ │ interval stream : ▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪──▪ │
+ │ events : START ─ Tput/Lat(k) ─ … ─ (RATE_STEP) ─ … ─ STOP │
+ ─────────────────────────────────────────────────────────────────────────
+
+
Figure 1 — One continuous run. Both the aggregate store and the interval stream cover the
+ whole run from t=0. Presentation tools may drop a startup window at read time (§10).
+
+
Integration point: Stats.AddLatency feeds both the aggregate store and the per-interval Welford
+ accumulator in one lock acquisition. Stats.TickInterval atomically snapshots and resets the
+ per-interval state plus a cumulative op-count. The Ticker goroutine calls
+ TickInterval on each tick and appends events to an in-memory EventBuffer; on
+ completion the harness attaches the buffer to the Result via Result.SetEvents.
+
+
+
+
4. Statistics & sample stores
+
+
Stats serializes all access under a mutex. It holds an aggregate SampleStore plus a
+ separate per-interval Welford accumulator that is always O(1) and independent of the chosen store mode.
+ All derived statistics are computed by the Result layer (stats.go): from the raw
+ Samples() in exact mode, and from the persisted histogram in hdr mode.
+
+
The SampleStore interface — benchkit/store.go
+
+// SampleStore accumulates latency samples (nanoseconds) for one run.
+// Implementations are not thread-safe; callers serialize access.
+// Derived statistics are computed by the Result layer: from Samples()
+// in exact mode, from the persisted histogram in hdr mode.
+type SampleStore interface {
+ Add(ns int64)
+ Count() uint64
+ Samples() []int64 // raw samples; nil in hdr mode
+ Reset()
+}
+
+
Stats modes
+
The aggregate store mode is the proto StatsMode enum (§11) directly —
+ there is no separate Go-only store-mode type.
+
+
+
+ Mode (StatsMode) |
+ Memory |
+ Provides |
+
+
+
+
+ StatsMode_EXACT (default) |
+ O(n) — 8 B/op |
+ Exact percentiles and the full distribution (raw samples on the Result). |
+
+
+ StatsMode_HDR |
+ O(1) — ~216 KiB |
+ Approximate percentiles, mean, and stddev from a log-linear histogram
+ (benchkit.Histogram, three significant figures ≈ 0.1% relative error); no raw
+ samples (Samples() is nil). The occupied buckets are persisted on the
+ Result as a LatencyHistogram (§11). |
+
+
+
+
The hdr store is an in-house implementation of the HdrHistogram design (benchkit/hdr.go):
+ bucket widths grow with the value's magnitude, resolving every sample to the configured significant
+ figures in O(1) record time and constant memory. Its API mirrors the common HdrHistogram bindings
+ (RecordValue, ValueAtQuantile, Mean, StdDev,
+ TotalCount, Min, Max), so switching to a library implementation
+ later is mechanical. The layout is fixed at nanosecond resolution up to a one-minute ceiling (clamped) —
+ a per-op latency near the ceiling would already have hit the benchSlack timeout.
+
newSampleStore(mode) returns an hdrStore for StatsMode_HDR and an
+ exactStore otherwise. The &Stats{} zero value lazily initializes to
+ StatsMode_EXACT on first use, so direct &Stats{} call sites get exact
+ behavior. Only the client-side store is mode-selectable: the Control server's
+ Stats is always exact, so server-measured latency samples are unaffected by
+ -stats-mode.
+
+
Construction and the interval accumulator
+
+func NewStats(mode StatsMode) *Stats
+
+func (s *Stats) AddLatency(l time.Duration) {
+ ns := l.Nanoseconds()
+ s.mu.Lock()
+ s.sampleStore().Add(ns) // aggregate store (exact / hdr)
+ s.intervalUpdate(ns) // per-interval Welford (always O(1))
+ s.mu.Unlock()
+}
+
+// TickInterval atomically snapshots and resets the per-interval Welford
+// accumulator and the op-delta counter. Called by the Ticker each tick.
+func (s *Stats) TickInterval() (mean, stddev float64, count, opDelta uint64)
+
Other Stats methods: Start/End bracket the measurement window;
+ AddOp increments the op counter without recording latency (server-measured sends);
+ AddLatencyBySender buckets samples per sender for clock-offset correction;
+ Ops returns the operations recorded since the last Clear;
+ GetResult / GetResultCorrected build a Result from the accumulated
+ state.
+
+
+
+
5. Event stream
+
+
The interval event stream is embedded in the result: each per-benchmark Result carries a
+ repeated events field, and the per-node Report container (written once per node)
+ holds them all. Because the event stream shares the aggregate side's per-benchmark, write-once lifecycle, a
+ binary whose -benchmarks regexp matches several benchmarks keeps each benchmark's event data
+ separate.
+
+
Event messages — benchkit/benchkit.proto
+
+// ThroughputInterval: ops-completed count and elapsed time for one tick.
+message ThroughputInterval {
+ uint64 ops = 1; // operations completed during this interval
+ int64 duration = 2; // actual elapsed interval, nanoseconds
+}
+
+// LatencyInterval: the Welford accumulator state for one tick.
+message LatencyInterval {
+ double mean = 1; // mean latency, nanoseconds
+ double stddev = 2; // sample stddev, nanoseconds
+ uint64 count = 3; // samples in this interval
+}
+
+// PhaseMarker: a lifecycle transition.
+message PhaseMarker {
+ enum Phase {
+ START = 0; // t=0; rate carries the initial target ops/s (default)
+ RATE_STEP = 1; // rate ramp step; rate carries the new target
+ STOP = 2; // run finished
+ }
+ Phase phase = 1;
+ int64 rate = 2; // target ops/s at START and RATE_STEP; 0 = unlimited
+}
+
+// Event: one time-stamped entry. Field 15 is reserved for a
+// google.protobuf.Any escape hatch (protocol-specific events).
+message Event {
+ int64 offset = 1; // nanoseconds since START (monotonic)
+ oneof payload {
+ ThroughputInterval throughput = 2;
+ LatencyInterval latency = 3;
+ PhaseMarker phase = 4;
+ // 15 reserved for google.protobuf.Any extension
+ }
+}
+
The oneof over the three well-known event types keeps the binary encoding compact and lets
+ consumers type-switch exhaustively. Field 15 is reserved (in a comment) for a
+ google.protobuf.Any escape hatch should a protocol author need to emit a custom event without
+ changing the shared schema.
+
+
The event buffer — benchkit/event_buffer.go
+
The buffer is a nil-safe in-memory accumulator: all methods are no-ops on a nil receiver. The
+ Ticker owns the buffer (§6) and is its only emitter; the type and its
+ emit methods are unexported (consumers that need synthetic event streams construct Event
+ values directly via the generated builders, as the timeseries tests do):
+
+func newEventBuffer() *eventBuffer
+func (b *eventBuffer) emitPhase(now time.Time, phase PhaseMarker_Phase, rate int64)
+func (b *eventBuffer) emitThroughput(now time.Time, ops uint64, duration time.Duration)
+func (b *eventBuffer) emitLatency(now time.Time, mean, stddev float64, count uint64)
+func (b *eventBuffer) Events() []*Event // nil on a nil receiver
+
The monotonic-offset base is set by the START phase marker (or the first emit); negative
+ offsets clamp to zero.
+
+
+
+
6. Ticker
+
+
The Ticker (benchkit/ticker.go) drives the interval stream. It ticks at the
+ configured interval, snapshots the per-interval Welford state from Stats, emits the interval
+ events to the EventBuffer it owns, and accumulates a second-level Welford over per-interval
+ throughput for a whole-run coefficient of variation.
+
+// NewTicker allocates an EventBuffer when interval > 0; interval == 0
+// disables event collection entirely (no buffer, no goroutine).
+func NewTicker(interval time.Duration, stats *Stats) *Ticker
+
+// Start emits START (carrying the initial rate) and spawns the goroutine
+// when interval > 0.
+func (t *Ticker) Start(rate int64)
+
+// RateStep emits RATE_STEP with the new offered load (rate ramping, §7).
+func (t *Ticker) RateStep(rate int64)
+
+// Stop stops the goroutine, emits STOP, and returns the whole-run
+// throughput CV (0 if fewer than two intervals).
+func (t *Ticker) Stop() (throughputCV float64)
+
+// Events returns the buffered events for Result.SetEvents; nil when
+// interval == 0.
+func (t *Ticker) Events() []*Event
+
The lifecycle is Start → (RateStep)* → Stop. On each tick the goroutine snapshots
+ TickInterval, computes interval throughput = opDelta / elapsed, emits a
+ ThroughputInterval, and emits a LatencyInterval only when the interval recorded at
+ least one sample. On shutdown the goroutine flushes the partial interval between the last tick and
+ Stop (an empty tail emits nothing), so the summed interval ops always equal the recorded ops;
+ the tail interval can be much shorter than the tick interval, so consumers derive per-interval throughput
+ from the recorded duration, never the configured interval. The whole-run CV returned by
+ Stop is available for logging; the persisted time series lets consumers compute a CV over any
+ window they choose (§10).
+
+
+
+
7. Rate ramping
+
+
The harness can step the offered load within a single run, tracing a throughput/latency saturation curve in
+ one execution. Combined with the interval time series, each rate step becomes a labelled stretch of
+ intervals that plot's SaturationCurvePlotter reads.
+
+
Ramping is controlled by two Options fields and is active only when both are set:
+
+type Options struct {
+ // … shared fields …
+ RateStep int // ops/s added per step; 0 = disabled
+ RateStepMax int // ceiling ops/s; 0 = disabled
+}
+
The step duration is derived, not configured: the ramp has one offered-load level per
+ RateStep increment from the start rate up to and including RateStepMax
+ (a partial final increment still counts as a level), and Duration is divided evenly across
+ the levels. A ramp therefore always spans exactly the configured duration and ends exactly at the
+ configured ceiling — there is no way to configure a ramp that idles at the ceiling or truncates before
+ reaching it.
+
runMeasure (in harness.go) implements the loop. When ramping is disabled it runs
+ the whole Duration at opts.Rate via a single RunPhase. When enabled
+ it runs one phase per level, calling ticker.RateStep(newRate) before each transition. The
+ ramp starts at Rate when set, and at RateStep when Rate is unset,
+ so the offered load always climbs from the first step; the START marker carries the actual starting rate.
+ The aggregate Result spans the full run across all steps; per-step detail lives in the event
+ stream.
+
+
+
+
8. Run harness
+
+
A benchmark is a named, runnable Bench. The harness Run stamps the shared run
+ metadata (RunConfig) onto each returned Result, so a Bench.Run
+ closure only fills in the measured fields.
+
+type Bench struct {
+ Name string
+ Description string
+ Run func(Options) (*Result, error)
+}
+
+// Run selects benches matching sel, runs each with opts, stamps RunConfig
+// metadata, and returns results sorted by benchmark name.
+func Run(sel *regexp.Regexp, opts Options, benches []Bench) ([]*Result, error)
+
+
The Measurement lifecycle helper
+
Every runner wires up observability through one helper
+ (benchkit/measurement.go), so all of them honor -stats-mode (the aggregate store)
+ and -interval (the event stream) identically:
+
+// StartMeasurement creates Stats + Ticker, emits START at opts.Rate, and
+// starts the clock. Record samples through the exported Stats field.
+func StartMeasurement(opts Options) *Measurement
+
+// Finish ends the run and returns the client-measured Result (built from
+// Stats) with the event stream attached.
+func (m *Measurement) Finish() *Result
+
+// Attach ends the run and attaches the event stream to a Result built
+// elsewhere (server-measured paths). The client Stats then carries only the
+// op count, via Stats.AddOp, for the throughput time-series.
+func (m *Measurement) Attach(result *Result)
+
Finish and Attach also record the run's MeasurementMode
+ (client-measured vs server-measured) on the result, so downstream consumers never infer it.
+
+
Reusable lifecycles
+
Two helpers build a Bench.Run closure for the common measurement shapes; both wrap a measurement
+ core that owns just the paced send window, leaving the control-plane lifecycle around it to the
+ helper. Both accept optional LifecycleOption hooks, and a hook error fails the run:
+ WithQuiesce registers a drain hook invoked after the measurement window closes and before
+ Control.Stop collects the server-side statistics, so in-flight operations are observed by
+ the boundary measurement; WithVerify registers a correctness check over the per-server
+ Control.Stop replies (nil in a local client-measured run, where there are no remote
+ replies).
+
+
+
+ | Helper |
+ Lifecycle |
+ Used by |
+
+
+
+
+ ClientMeasured |
+ StartRemote → MeasureLatency (paced client-timed window from t=0) →
+ StopRemote to collect server memory stats. |
+ QuorumCall, SlowServer |
+
+
+ ServerMeasured |
+ Control.Start → MeasureOneWay (paced one-way send window) wrapped in
+ RunOffsetCorrected for clock correction → Control.Stop to collect
+ server latency samples. |
+ Multicast |
+
+
+
+
ServerMeasured reports client-side memory cost per send: it divides the client's
+ memory delta by the client's own op count (Stats.Ops), not by the aggregated server op count,
+ which for multicast would be N× the sends and under-report the per-send cost.
+
+
Measurement cores
+
The two cores are the reusable send window underneath the helpers. Each takes the run configuration(s) and a
+ per-configuration setup/send closure and drives runMeasure (so both inherit rate
+ ramping and the ticker stream); yielding one configuration drives the coordinator case and yielding one per
+ peer system drives the symmetric case in a single shared phase:
+
+ MeasureLatency- Client-measured core. Builds a timed doOp per configuration that records each op via
+
Stats.AddLatency, runs the window, and returns the Finished
+ Result. Backs ClientMeasured (one config) and
+ runSymmetricQuorumCall (one config per system).
+ MeasureOneWay- Server-measured core. Wraps each one-way send into a doOp that records a client op via
+
Stats.AddOp (the op count feeds only the throughput series; latency is measured server
+ side) and returns the Measurement plus a window closure for the caller to feed into
+ RunOffsetCorrected and then Attach. Backs ServerMeasured (one
+ send) and runSymmetricMulticast (one send per system).
+
+
+
Shared primitives
+
+ StartRemote / StopRemote- The client-measured control-plane lifecycle:
StartRemote resets the remote servers'
+ counters and StopRemote collects their per-op memory stats into the result. Both are
+ no-ops in local mode, where in-process server memstats cannot be separated from the client. Shared by
+ ClientMeasured and runAsyncQCBenchmark.
+ RunPhase- Launches
numG paced worker goroutines per doOp in one errgroup until
+ endTime; multiple doOps fan out concurrent send targets (the symmetric
+ runners).
+ RunOffsetCorrected- Estimate offsets → measure → estimate again → build per-node corrected results → aggregate. Generic over
+ the offset type so it serves both the coordinator and symmetric callers. The aggregation
+ (
AggregateServerResults) sums ops and throughput and concatenates samples, but takes the
+ maximum total_time across servers — the measurement window's wall-clock time, not
+ a sum.
+ BenchContext- A context bounded by
Duration + benchSlack so a stuck RPC fails rather than hangs.
+ CollectReplies- Drains a
ResponseSeq into a per-node map, joining every node's error rather than dropping
+ failures.
+
+
+
The custom runner runAsyncQCBenchmark (in benchmark/benchmark.go) does not fit the
+ window cores — it fires from completion callbacks rather than a fixed worker pool, paced by a shared
+ RatedGate — but still uses StartMeasurement plus StartRemote /
+ StopRemote, so every runner honors -interval and -stats-mode and
+ emits a time-series stream. The symmetric runners are thin wrappers over the cores:
+ runSymmetricQuorumCall over MeasureLatency and
+ runSymmetricMulticast over MeasureOneWay.
+
+
+
+
9. Flag contract
+
+
StandardFlags (benchkit/flags.go) is the CLI contract every sweep-driven benchmark
+ binary accepts; RegisterFlags registers it and Options() maps it to run
+ Options. A protocol binary adds its own flags to the same FlagSet.
+
+
+ | Flag | Default | Meaning |
|---|
+
+
+ -benchmarks | .* | Regexp selecting benchmarks to run. When invoked directly this is a full Go regexp (QuorumCall matches all variants as a substring; use ^QuorumCall$ for an exact match). cmd/sweep passes a comma-separated list of exact names and automatically anchors each as ^name$ — never use | as a separator there. |
+ -self | — | This node's listen address; non-empty triggers distributed mode. |
+ -remotes | — | Comma-separated peer addresses. |
+ -workers | 1 | Concurrent worker goroutines. |
+ -payload | 0 | Request/response payload size, bytes. |
+ -rate | 0 | Target sends/s per node; 0 = unlimited (saturating). |
+ -time | 1s | Measurement duration. |
+ -output | — | Result file path. |
+ -verbose | false | Log connection progress. |
+ -stats-mode | exact | exact | hdr aggregate store. |
+ -interval | 500ms | Ticker interval; 0 disables events. |
+ -rate-step | 0 | Rate increment per ramp step; 0 = disabled. |
+ -rate-step-max | 0 | Ceiling ops/s for the ramp; 0 = disabled. Step duration is derived (§7). |
+ -stream-mode | dual | Symmetric stream topology: dual (dedicated outbound streams) or dedup (reuse inbound peer streams via WithStreamDedup). |
+ -call-timeout | 0 | Per-call deadline for quorum-call workloads; a call stalled behind an unresponsive peer fails with DeadlineExceeded instead of hanging until run end. 0 = disabled. |
+ -cpuprofile | — | CPU profile output path; empty = disabled. |
+ -memprofile | — | Heap profile output path; empty = disabled. |
+ -trace | — | Execution trace output path; empty = disabled. |
+ -fault-kill-after | 0 | Fault injection: exit cleanly after this duration; 0 = disabled. |
+
+
+
The profiling flags feed benchkit.StartProfilers (benchkit/profiling.go), so
+ every binary built on the contract can produce profiles without extra wiring.
+ cmd/benchmark adds gorums-specific extras on top of the contract: -config-size,
+ -quorum-size, -max-async, -send-buffer/-recv-buffer,
+ -server, -server-stats, -list, -label, and
+ -compare.
+
+
How the flags interact
+
+
-workers × -rate
+
-workers caps concurrency (the number of in-flight operations); -rate sets the
+ offered load. With -rate 0 the run is closed-loop: every worker fires its next op
+ as soon as the previous one completes, so throughput ≈ workers / latency and the system saturates. With
+ -rate > 0 the run is open-loop: the rate is split equally across the workers,
+ each pacing its sends on an absolute schedule (Pacer), staggered so sends do not burst.
+
The two are coupled through latency: sustaining rate R with per-op latency L
+ requires roughly workers ≥ R × L in-flight slots. If workers are too few, the pacers fall
+ behind their schedules and the run degrades to closed-loop saturation while the START marker still
+ claims the offered rate. The harness detects this: a paced run that attempts fewer than 95% of its
+ scheduled sends prints a warning on stderr naming the worker-sizing fix. Size -workers
+ for the worst-case latency of the highest rate (including the top of a ramp).
+
+
+
-stats-mode × -interval
+
The two are independent recorders of the same samples: -stats-mode picks the aggregate
+ store for the whole run, -interval controls the time-resolved event stream. Every mode is
+ self-sufficient — an hdr run carries its whole-run distribution in the histogram even with
+ -interval 0 — so the flags compose freely. The only coupling is at read time: hdr runs
+ have no raw samples, so trimmed statistics come from the interval events
+ (§10); without events an hdr run reports whole-run numbers only.
+
+
-fault-kill-after arms benchkit.ArmFaultInjection, which schedules a clean
+ os.Exit(0) via time.AfterFunc — a crash-stop fault, not an error. Under sweep,
+ the dead node's missing result file is logged as a warning and the summary reports the surviving nodes; to
+ kill only a subset of nodes, run the doomed node(s) manually or pass the flag via
+ -extra-args (which applies to every node).
+
+
+
+
10. Read-time trimming
+
+
The binary records the whole run; consumers may compute statistics over a trimmed view that excludes the
+ startup transient. The mechanism is an index map from the event stream: the raw latency
+ samples are not individually time-stamped, but the event stream is, so a time threshold maps to a
+ sample-index cut via the cumulative per-interval op counts.
+
+
A consumer chooses a trim threshold T (e.g. -trim 1s; default 0 = no
+ trim), walks the events summing ThroughputInterval.ops for intervals with
+ offset < T, and uses that sum as the sample-index cut k. Summary statistics are
+ recomputed over latencies[k:] and the time-series plots drop intervals before T.
+ The mechanism is exported as benchkit.Summarize(r *Result, trim time.Duration) Summary
+ (benchkit/summary.go), which every consumer calls, sweep included.
+
+
+
Validity by measurement style
+
Client-measured benchmarks call AddLatency exactly once per completed op,
+ in completion order, so the sample index equals the cumulative op count and the cut is exact.
+ Server-measured benchmarks increment the op counter via AddOp while
+ latency samples arrive server-side and are clock-corrected afterwards; ops and samples are not 1:1, so
+ the index map does not apply and these runs trim at interval granularity only (mean/stddev/throughput
+ from the event stream; whole-run percentiles). In hdr mode there are no raw samples and
+ the histogram has no time dimension, so throughput trims at interval granularity while the
+ distribution stays whole-run.
+
Summarize reads RunConfig.measurement_mode and stats_mode
+ (§11) and applies the index-map cut only when the run is
+ CLIENT_MEASURED and EXACT.
+
+
The throughput coefficient of variation is derived at read time over the trimmed interval throughputs;
+ there is no throughput_cv field in the schema, so the CV in a summary row is always consistent
+ with the trim window used for the other columns of that row.
+
+
+
+
11. Proto schema
+
+
The schema lives in a single file, benchkit/benchkit.proto. Configuration metadata is grouped
+ in a RunConfig message; the measured fields stay flat on Result; the event stream
+ rides inside Result. Duration/offset fields carry the unit in the comment, not a
+ _ns suffix.
+
+
+edition = "2024";
+package benchkit;
+option features.field_presence = IMPLICIT;
+option go_package = "github.com/relab/gorums/benchkit";
+
+// MeasurementMode: who timed each op (decides index-map trim validity).
+// StatsMode: which aggregate store (exact has raw samples; hdr a histogram).
+// Zero value is a meaningful default (client-measured, exact); no UNSPECIFIED.
+enum MeasurementMode { CLIENT_MEASURED = 0; SERVER_MEASURED = 1; }
+enum StatsMode { EXACT = 0; HDR = 2; reserved 1; } // 1 was WELFORD; removed
+
+// RunConfig: configuration metadata for one run.
+message RunConfig {
+ string name = 1; // benchmark name, e.g. "QuorumCall"
+ int32 num_nodes = 2; // nodes in the configuration
+ string mode = 3; // "local" or "remote"
+ int64 duration = 4; // configured run duration, nanoseconds
+ int32 workers = 5; // concurrent worker goroutines
+ int32 payload = 6; // payload size, bytes
+ int64 rate = 7; // target ops/s per node; 0 = unlimited
+ int64 interval = 8; // ticker interval, nanoseconds; 0 = events disabled
+ MeasurementMode measurement_mode = 9; // client- vs server-measured
+ StatsMode stats_mode = 10; // exact vs hdr aggregate store
+ string stream_mode = 11; // "dual" or "dedup" stream topology
+ int32 quorum_size = 12; // replies awaited per quorum call; 0 if N/A
+ int32 max_async = 13; // max in-flight async calls; 0 if N/A
+ int64 rate_step = 14; // ramp increment, ops/s; 0 = no ramp
+ int64 rate_step_max = 15; // ramp ceiling, ops/s; 0 = no ramp
+ int64 call_timeout = 16; // per-call deadline, nanoseconds; 0 = disabled
+}
+
+// MemoryStat: per-server memory statistics.
+message MemoryStat { uint64 allocs = 1; uint64 memory = 2; }
+
+// Result: one benchmark's complete output for one node.
+message Result {
+ RunConfig config = 1; // how the run was configured
+
+ // Aggregate measured results over the whole run.
+ uint64 total_ops = 2;
+ int64 total_time = 3; // elapsed wall time, nanoseconds
+ double throughput = 4; // ops/s over the whole run
+ uint64 allocs_per_op = 5;
+ uint64 mem_per_op = 6; // bytes per operation
+ repeated MemoryStat server_stats = 7;
+
+ // Raw per-op latency samples, nanoseconds; nil in hdr mode.
+ // Signed: clock-offset correction can yield negatives.
+ repeated int64 latencies = 8;
+
+ // Time series covering the whole run; empty when interval = 0.
+ repeated Event events = 9;
+
+ // Latency distribution for StatsMode_HDR runs; nil otherwise.
+ LatencyHistogram histogram = 10;
+
+ // Field 11 held the retired StreamStats message (stream-topology
+ // statistics); reserved so older result files never fail to decode.
+ reserved 11;
+
+ // Ops that returned an error and were counted but not aborted on
+ // (client-measured runs). Total attempts = total_ops + failed_ops.
+ uint64 failed_ops = 12;
+}
+
+// LatencyHistogram: the bounded-memory distribution of an hdr run, as
+// ascending weighted (value, count) pairs — value is the bucket's
+// median-equivalent latency in nanoseconds. Consumers treat the pairs as
+// a weighted sample set: quantiles, mean, and stddev are computed over
+// them without knowing the producer's bucket layout.
+message LatencyHistogram {
+ repeated int64 value = 1;
+ repeated uint64 count = 2;
+}
+
+// Report: the per-node container, written once per node.
+message Report {
+ string label = 1; // "baseline" / "experiment"
+ repeated Result results = 2; // one per matched benchmark
+}
+
+// Event / ThroughputInterval / LatencyInterval / PhaseMarker: see §5.
+
+
RunConfig is separable "how the run was set up" metadata, read via one hop
+ (r.GetConfig().GetName()); measured fields are the primary content of a Result
+ and stay flat (r.GetThroughput(), r.GetLatencies(),
+ r.GetEvents()).
+
+
The same file also defines the reduced schema a consumer writes after aggregating a whole sweep —
+ PlotData, PlotRun, PlotBenchmark, PlotNode, and
+ LatencySummary (§14). A producer never writes these; they live here so
+ the reduction reuses RunConfig for a run's dimensions instead of restating them, and so a
+ consumer compiles one schema rather than two overlapping ones.
+
+
+
+
12. Result file format & contract
+
+
This section is the contract between a benchmark binary (the producer, in any language) and the tools that
+ read its results (sweep, plot, or any other consumer). Stabilizing it is what
+ lets sweep summarize any protocol's run without importing the producer's code. The proto
+ messages themselves (§11) are the single source of truth for field numbers, types,
+ and comments; this section covers what the proto cannot express — framing, conversion, aggregation, and
+ producer/consumer obligations.
+
+
On-disk encoding
+
Each node writes one binary file (benchkit.WriteReport / LoadReport /
+ DecodeReport in report.go):
+
+[8 bytes] magic sentinel "BKRSv2\n\x00" (0x42 4B 52 53 76 32 0A 00)
+[remaining] binary-encoded Report message (proto.Marshal)
+
The sentinel exists for identification, not versioning: proto.Unmarshal accepts
+ almost any byte string without error, so the sentinel is what lets a consumer reject a non-benchkit (or
+ truncated) file cleanly instead of decoding garbage into a zero-valued Report. The
+ v2 digits bump only if the framing itself changes (e.g. a length-delimited stream of
+ messages), which protobuf wire compatibility cannot absorb. The sentinel is defined once, in
+ benchkit/report.go. A consumer that reads the file itself — sweep skips absent
+ files rather than failing on them — decodes the bytes with DecodeReport instead of
+ re-declaring the sentinel, so producer and consumer cannot drift apart.
+
+
Schema sharing and evolution
+
The file embeds no descriptor. benchkit/benchkit.proto is a small, gorums-free file (no
+ imports), so a consumer outside this repository can compile it standalone with plain
+ protoc-gen-go into its own package, sharing only the schema, not the code. Consumers inside
+ the benchkit module, sweep included, import the generated types directly: two
+ descriptor sets registering the same file names in one binary would panic at init. The schema
+ evolves under standard protobuf wire-compatibility rules with no version bump: add fields with new, never-reused
+ numbers; consumers ignore fields they were not compiled with, and absent fields read back as zero. A
+ protocol may define its own result message that reuses the common field names and numbers.
+
+
Converting to protojson
+
Local sweep runs write a protojson .json sibling next to each collected
+ .binpb file for manual inspection; the magic sentinel is not
+ carried into the .json. Driver runs instead reduce successful .binpb files to a
+ compact, normalized plotdata/plotdata.binpb on the driver and transfer that file by default,
+ while failed-run .binpb files are still downloaded for diagnosis. Note that protojson encodes
+ int64/uint64 fields (including latencies) as JSON strings, not
+ numbers, so a consumer parses them from strings into integers; throughput is a
+ double and stays a JSON number.
+
+
Producer and consumer obligations
+
A producer writes one Report per node, with label identifying the run or node,
+ and populates per benchmark at least config.name, throughput (per node), and
+ latencies (nanoseconds; nil when the stats mode keeps no raw samples — check
+ RunConfig.stats_mode), filling metadata and event fields when known. Those three fields are
+ also all a summarizing consumer needs; everything else may be ignored. A consumer aggregates a run's
+ per-node files as follows: throughput is reported per node and summed across nodes;
+ latency percentiles are recomputed from the merged sample set across all nodes, never averaged
+ from per-node percentiles. The startup transient is not removed by the producer; consumers trim at read
+ time (§10). A producer must also write all diagnostic output to stderr
+ (in-module code uses benchkit.Logf): a launcher that leaves stdout unread lets the SSH
+ channel's flow-control window fill and block the process (see
+ troubleshooting §4.6).
+
+
report.go also provides PrintComparison for the -compare path
+ (baseline vs experiment, percentage deltas).
+
+
+
+
13. sweep
+
+
cmd/sweep runs the Cartesian product of -n × -workers × -payload × -rate ×
+ -send-buffer × -recv-buffer × -benchmarks × -stream-mode × -reps over a set of SSH hosts. For each run it kills lingering processes, launches one
+ benchmark binary per node, waits, downloads the per-node .binpb result files for local runs,
+ writes a protojson .json sibling for each (for inspection), and prints a
+ summary table. In driver mode, successful runs are reduced to compact plot CSVs on the driver; failed-run
+ .binpb files are downloaded with the compact results. Per-run output files are named
+ <label>_<bench>_N<n>_W<w>_P<p>[_R<rate>][_SB<send>][_RB<recv>]_S<stream>_r<rep>_<host>_<port>.
+ The _SB and _RB components appear only when that buffer is swept.
+
+
Flag reference
+
+
+ | Flag | Default | Meaning |
|---|
+
+
+ | Swept dimensions — comma-separated lists; the Cartesian product defines the runs |
+ -n | 9 | Node counts to sweep. |
+ -workers | 1 | Worker counts to sweep. |
+ -payload | 0 | Payload sizes (bytes) to sweep. |
+ -rate | 0 | Target sends/s per node to sweep; 0 = unlimited (saturating). |
+ -send-buffer | binary default | Per-node send queue capacities to sweep. Unset leaves the binary's own default and adds no dimension. |
+ -recv-buffer | binary default | Server receive queue capacities to sweep. Unset leaves the binary's own default and adds no dimension. A capacity of 0 is a real setting (unbuffered), distinct from unset. |
+ -benchmarks | SymmetricQuorumCall | Exact benchmark names (each anchored as ^name$; not a regexp — see §9). |
+ -stream-mode | dual | Symmetric stream topology to sweep: dual or dedup. The default dual mode preserves existing behavior. The special baseline mode labels runs of a prebuilt pre-dedup binary (requires -binary, cannot be mixed with other modes, and is never passed to the node command line). |
+ -reps | 1 | Repetitions per parameter combination. |
+ | Run setup |
+ -hosts | — | SSH host aliases: bb[1-30] ranges, globs, or comma-separated literals. Required except with -collect or a local -explain-check. |
+ -config | ~/.ssh/config | SSH config file used to resolve -hosts aliases. |
+ -port | 9000 | Base port for benchmark nodes (extra nodes on a host use successive ports). |
+ -duration | 10s | Measurement duration per run (forwarded as the binary's -time). |
+ -outdir | out | Root output directory for sweep runs. |
+ -sweep | run | Label prefix for output filenames and, when set explicitly, the run directory name. |
+ -binary / -build | auto-build | Pre-built linux/amd64 binary, or a build command with the {{output}} token (also via $BENCHKIT_BUILD); default go build ./cmd/benchmark. |
+ -fd-limit | 65536 | Soft open-file limit (ulimit -Sn) applied to every node and the driven sweep; 0 keeps the host default. |
+ -test | 0 | Quick smoke test with N nodes (5 s, one worker, defaults elsewhere); 0 = full sweep. |
+ -verbose | false | Pass -verbose to the nodes. |
+ | Health checks & summarizing |
+ -check | false | Run host diagnostics and exit (see Diagnostics below). |
+ -netcheck | true | Ping-ring probe of every host link before the sweep; aborts on ≥ 5% packet loss. |
+ -degraded-below | 0.5 | Flag a run degraded when a node's throughput falls below this fraction of the run median (0 disables); flagged nodes, the bound each crossed, and per-host TCP counter deltas are recorded in the manifest. |
+ -degraded-above | 2 | Flag a run degraded when a node's throughput exceeds this multiple of the run median (0 disables). A symmetric benchmark cannot have one node doing several times its peers' work, so this is the signature of operations recorded without a network round trip. |
+ -degraded-latency-below | 0.2 | Flag a run degraded when a node's median latency falls below this fraction of the run median (0 disables) — independent evidence of the same anomaly: a quorum call completing in a fraction of its peers' time did not do the round trip. |
+ -trim | 0 | Read-time trim for the summary table and manifest (§10); never forwarded to nodes. |
+ | Pass-through to every node — appended only when set, so a foreign binary lacking them keeps working as long as the sweep does not ask for them |
+ -interval, -stats-mode | binary default | Event stream and aggregate store (§9). |
+ -rate-step, -rate-step-max | 0 | Rate ramping (§7). |
+ -extra-args | — | Arbitrary string appended verbatim to every node command (protocol-specific flags of foreign binaries). |
+ -collect-profiles, -pgo | false | Profile collection and PGO merge (below). |
+ | Cluster-local driver (below) |
+ -driver | — | Run the orchestration on this cluster-local host (first = first -hosts entry, excluded from the pool). |
+ -detach | false | Start the driven run and exit once it is safe to disconnect; requires -driver. |
+ -collect [path] | latest | Collect a finished run once. With no path, use the driver and path in <outdir>/.sweep-last.json; an explicit path requires -driver. |
+ -collect-now [path] | latest | Collect a best-effort snapshot even while the run is active. Mutually exclusive with -collect. |
+ -list | false | List active, completed, raw-pending, and recoverable driver runs, newest first. |
+ -remote-dir | /tmp | Remote storage root; sweep uses <root>/sweep-$USER for binaries, results, caches, and driver runs. |
+ -transfer | rsync | Transfer backend for driver uploads/downloads: rsync ≥3.2.4 (default) or sftp. |
+ | LLM failure triage (§15) |
+ -explain, -explain-check | false | Triage failed runs after the sweep / verify the provider and exit. |
+ -explain-provider, -explain-model | local, — | Backend (local | openai | claude) and model (required with -explain). |
+ -explain-max-log | 65536 | Head+tail byte cap on the node log in the triage prompt. |
+
+
+
Three internal flags (-driven, -git-sha, -ready-marker)
+ are set automatically by the driver machinery and never passed by hand.
+
+
Host resolution and node identity
+
-hosts names SSH aliases, but the nodes never see them: sweep resolves each connected host
+ once, controller-side, to one stable non-loopback peer IP (rejecting loopback/unspecified/multicast,
+ preferring IPv4) and passes numeric ip:port values in -self and
+ -remotes. This removes DNS from remote process startup entirely — a transient resolver
+ failure on one host used to abort the whole run (see
+ troubleshooting §4.2). The alias is kept for
+ SSH, filenames, logs, and cleanup, and the manifest's node_map records the
+ alias↔address↔node-ID correspondence — needed because numeric-IP sorting can assign different Gorums
+ node IDs than hostname sorting would.
+
+
Profile collection and PGO
+
-collect-profiles passes per-node -cpuprofile/-memprofile paths to
+ every node (the standard profiling flags, §9) and downloads the resulting
+ *.cpu.prof/*.mem.prof files alongside the result files, with the same naming
+ scheme and remote cleanup. -pgo (implies -collect-profiles) merges every
+ collected CPU profile into <outdir>/default.pgo after the sweep — drop it into a main
+ package directory (or point go build -pgo at it) for profile-guided optimization. Execution
+ traces are deliberately not collected: a saturating multi-second run produces traces too large to ship
+ over the SSH path by default; pass -extra-args '-trace=…' and fetch manually when one is
+ needed.
+
+
Per-run manifest
+
Before launching a run's nodes, sweep writes <base>.manifest.json into the output
+ directory: the run configuration (label, benchmark filter, nodes, workers, payload, rate, stream mode, rep, duration,
+ trim), a timestamp, the repository HEAD (best effort), the deployed binary, the host:port assignments, and the
+ expected per-node result file names. The manifest is written up front so the directory is self-describing
+ even when a run fails, and it is the canonical way for consumers to group a run's per-node files —
+ the report generator discovers runs through it rather than parsing filenames. Manifest write
+ failures are logged and do not abort the run.
+
After the run, the manifest records the outcome: status, failure_phase
+ (setup / measurement / collection), collected_files / missing_files,
+ node_map, degraded-node flags with per-host TCP counter deltas, and the -explain
+ diagnosis when enabled. There is no automatic retry — a failed or degraded cell stays visible as such.
+ Field semantics and how to read them are covered in
+ troubleshooting §2, alongside the other per-run
+ artifacts (the interleaved node log logs/<base>.log and the failure snapshot
+ logs/<base>_snapshot.txt).
+
+
Summary table — cmd/sweep/summary.go
+
After each run, sweep loads every node's result file and prints an aggregated table: throughput summed
+ across nodes, latency percentiles recomputed from the merged samples. Missing or unreadable files are
+ skipped with a warning so a partial run still reports what it collected.
+
+ BENCHMARK THROUGHPUT CV MEAN STDDEV p50 p95 p99 NODES SAMPLES
+
The -trim flag is consumed entirely at read time — it is not forwarded to the nodes
+ (the binary always records the whole run, §10). It drives this summary table and is
+ recorded in the per-run manifest, where sweep -plot picks it up and applies the same cut
+ (§14), so the figures and the table describe the same steady-state window.
+ benchkit.Summarize
+ recomputes per-node throughput over the kept intervals, computes the throughput CV (σ/μ) over those
+ intervals, and cuts the latency slice at the sample index implied by the dropped intervals' cumulative op
+ counts — for client-measured exact runs, the case where the cut is valid. It returns a
+ Summary whose CVValid and LatencyValid flags tell
+ mergeResults which fields are meaningful. Each summary's Dist merges into one
+ benchkit.LatencyDist per benchmark, which answers the MEAN,
+ STDDEV, and percentile columns from the merged raw samples when any node contributed them
+ and from the merged LatencyHistogram weighted pairs otherwise (hdr runs) — whole-run
+ figures in that case, since the histogram has no time dimension (§10). When a result
+ carries no events, the stored whole-run throughput and latencies are used and the CV column shows
+ -.
+
+
Diagnostics
+
sweep -check exercises the same SSH path as a sweep and reports, per host: reachability, host
+ info (kernel, CPUs, load), busy benchmark ports, lingering benchmark processes, TCP retransmit health,
+ clock skew, remote storage/free space, and a concise count of disposable sweep artifacts left behind.
+ It never deletes those artifacts; driver-run details belong to -list. Run it before a long sweep. Independently, every sweep
+ starts with the -netcheck ping-ring probe (on by default) and aborts on ≥ 5% link loss —
+ note it runs on an idle network, so a link that only drops packets under load can pass it; the
+ -degraded-below flagging catches those after the fact (see
+ troubleshooting §5).
+
+
Cluster-local driver
+
By default sweep runs on the developer's laptop and does all SSH from there: it uploads the
+ benchmark binary to every host and performs per-run launch / monitor / collect round-trips. When the
+ laptop is far from the cluster (e.g. driving the Stavanger bb cluster from abroad), the
+ binary is copied across the WAN once per host and every round-trip pays the full WAN RTT.
+ The -driver flag moves the orchestration onto a cluster-local host so that traffic stays on
+ the LAN and the benchmark binary crosses the WAN only once:
+
+# Run the orchestration on a dedicated head node (outside -hosts):
+./cmd/sweep/sweep -hosts 'bb[1-9]' -driver bbhead -sweep e4 -n 9 -workers 1,2,4,8,16
+
+# Or use the first -hosts entry as the driver (excluded from the benchmark pool):
+./cmd/sweep/sweep -hosts 'bb[1-10]' -driver first -sweep e4 -n 9 -workers 1,2,4,8,16
+
The driver role lives in sweep, not the benchmark binary: the orchestration depends on
+ iago (SSH, upload, collection) and cmd/benchmark must stay free of that
+ dependency. The laptop cross-builds sweep and the benchmark binary for linux/amd64, ships
+ them plus a generated SSH config to the driver over the user's own ssh/scp,
+ and re-execs sweep there with the internal -driven flag. The driver then does
+ all the per-node SSH over the LAN. On completion, the laptop downloads the compact transfer directory:
+ manifests, logs, plotdata/plotdata.binpb, plotdata/events.binpb, and any
+ failed-run .binpb files.
+ Successful raw .binpb files remain in the driver work directory until explicitly collected.
+
At launch, sweep creates the concrete local output directory and writes sweep.sh,
+ collect.sh, and the latest-run pointer <outdir>/.sweep-last.json.
+ A plain sweep -collect checks once: if the run is active it exits with guidance instead of
+ waiting; sweep -collect-now downloads a snapshot and retains the remote run. After compact
+ collection, rerun sweep -collect to archive
+ the full raw .binpb dataset locally and remove the driver work directory. If the compact output
+ is enough, discard the remote work directory with the cleanup command printed by sweep.
+
-remote-dir selects the remote storage root (for example /local).
+ Every host uses a per-user namespace below it, such as /local/sweep-meling. Use
+ sweep -list to inspect driver runs without making -check output unbounded.
+
Authentication to the peers uses the laptop's SSH agent, forwarded to the driver with ssh -A.
+ iago authenticates via SSH_AUTH_SOCK and dials every peer once at startup, so the forwarded
+ agent is only needed for the first few seconds of a run. After that the remote sweep — which is launched
+ detached (setsid) — survives a laptop disconnect. By default the launcher streams
+ its output live and prints a reconnect command; if the connection drops, reconnect and download the
+ results with -driver <host> -collect <remote-work-dir>. When the driver host is
+ one of -hosts (the first form), it is excluded from the benchmark pool so the
+ orchestrator does not perturb a co-located replica; a driver outside -hosts leaves the pool
+ unchanged.
+
Add -detach to start the run and exit immediately instead of streaming and waiting:
+
+./cmd/sweep/sweep -hosts 'bb[1-9]' -driver bbhead -detach -sweep e4 -n 9 -workers 1,2,4,8,16
+
-detach prints the same reconnect command and then returns, so closing the laptop lid right
+ after is an intentional clean exit rather than a dropped connection that surfaces as an error. Reconnect
+ the same way as an accidentally dropped run, with
+ -driver <host> -collect <remote-work-dir> — the two paths differ only in what the
+ launcher does before the run finishes, not in how the detached run itself behaves or how it is collected.
+ -detach requires -driver and cannot be combined with -collect.
+
The driven sweep authenticates to its peers using the laptop's SSH agent, forwarded to the driver; it
+ needs that agent only once, for the one-time dial of every peer at startup. So -detach does
+ not exit the instant the run starts — it keeps the laptop connected until the driven sweep signals that
+ dial is complete (it touches a peers.dialed marker in the work dir), then reports that it is
+ safe to disconnect. If exit.code appears before that marker, the run died before
+ dialing — normally because a peer denied SSH agent-forwarded authentication — and -detach
+ fails loudly with the console log tail instead of telling you it is safe to disconnect from a dead run.
+ The wait is bounded by a generous backstop (minutes) to cover a slow -explain preflight and
+ dialing a large cluster; it normally resolves in seconds. Waiting for the dial rather than a fixed timeout
+ is what makes -detach reliable regardless of cluster size: leaving before the dial completes
+ tears down the forwarded agent and the run fails with "no valid authentication methods".
+
+
Why system ssh/scp for the laptop↔driver hop, not iago?
+
The driver must authenticate to the peers using the laptop's keys, which means the laptop→driver
+ connection has to forward the SSH agent. iago has no agent-forwarding support (it authenticates the
+ laptop's connection from SSH_AUTH_SOCK but never requests forwarding for the
+ remote session), so the control hop uses the system ssh -A / scp the user
+ already relies on (ssh bbN), which also inherits their working SSH config and
+ host-key state for free. iago is still used for everything on the driver (driver→peer
+ dial, upload, collection). The natural follow-up is to add agent forwarding to iago (a
+ RequestAgentForwarding option on the session); the laptop↔driver upload/download
+ could then move to iago.Upload/iago.Download against a one-host group, and
+ the launcher would no longer shell out. Tracked as an iago enhancement.
+
+
Mechanism note: detaching and following the run is a small shell bootstrap —
+ setsid puts the driven sweep in its own session (a dropped connection cannot
+ SIGHUP it), output is redirected to a log file, an exit-code sentinel marks completion,
+ and the launcher tail -Fs the log until the sentinel appears. It needs nothing beyond
+ coreutils; tmux, systemd-run, or a Go-based daemon were considered and deferred as added
+ dependencies or added code for the same behavior.
+
+
Requirements
+
Plain ssh <host> must work from the laptop to the driver and from the driver to
+ each peer (the bbchain cluster resolves peers by hostname and permits inter-host SSH), and the
+ laptop's SSH agent must hold the key the peers accept. The driver needs bash,
+ setsid, tar, and tail (standard on Linux). The generated
+ config sets StrictHostKeyChecking no for the trusted cluster LAN, so the driver needs
+ no seeded known_hosts.
+
+
+
+
+
14. Plotting
+
+
sweep -plot <dir> turns a collected sweep directory into a self-contained Typst report.
+ It reads the compact plotdata/plotdata.binpb when present (the driver transfer), falls back
+ to the legacy plotdata/*.csv pair for directories collected before that format existed, and
+ falls back further to decoding the raw .binpb result files (a local sweep with no compact
+ data). It writes a set of derived CSVs and a report.typ under <dir>/report,
+ copies the embedded helper library gorumsplot.typ beside it, and compiles
+ report.pdf when the typst binary is on PATH (otherwise it prints
+ the compile command). A local sweep and a driver collection run this automatically; re-run it by hand to
+ regenerate the report with different filters. Typst never reads plotdata.binpb directly —
+ only the small, per-figure CSVs under report/ — so growing the sweep or the CDF resolution
+ does not slow down the Typst compile step.
+
+
plotdata.binpb
+
plotdata/plotdata.binpb is a normalized PlotData message, defined in
+ benchkit/benchkit.proto alongside the raw result schema and so generated with it: run
+ identity (label, status, repetition) is stored once
+ per run and node identity once per node, instead of being repeated on every row the way the old CSV pair
+ did — a node's 200-point latency CDF is a single packed double vector, with each point's
+ cumulative probability implied by its position on the fixed grid rather than stored. The benchmark name
+ and sweep dimensions ride in a RunConfig, the same message the raw result files carry them
+ in, so there is one definition of a run's configuration rather than a parallel one for plotting. This
+ keeps the file small (tens of MB rather than hundreds) and fast to load, both across the network and when
+ -plot reads it back. Because PlotData has only one repeated field, two
+ plotdata.binpb files can be combined with a plain cat, the same as the old CSV
+ pair (see the merged-sweep recipe below). Run sweep -export-csv <dir> to regenerate a
+ human- and grep-friendly plotdata/runs.csv and plotdata/nodes.csv from it on
+ demand — nodes.csv has one row per node with its CDF as a space-separated vector column,
+ rather than one row per CDF point.
+
+
events.binpb
+
plotdata/events.binpb is a PlotEvents message holding the time-series event
+ stream of every run, whatever its outcome, nested base → benchmark → node so a node is named
+ once however many events it recorded. It is the source of the within-run time-series figures. The streams
+ live in their own file rather than inside PlotData so the summary-only read paths
+ (-plot's aggregates, -export-csv, -explain) keep reading the small
+ message, and a directory collected before the streams existed stays readable. A stream is a small
+ fraction of a raw result file — the bulk of that file is latencies, which the report never
+ reads — so keeping all of them costs tens of megabytes for a sweep of several hundred runs, and the
+ compact transfer no longer has to leave the time-series data behind on the driver.
+
+
The report covers two layers from one command:
+
+ - Cross-run figures: how throughput, latency, goodput, and per-op cost change as a sweep
+ dimension (nodes, workers, payload, rate, stream mode) varies, plus throughput-latency curves,
+ per-node latency CDFs, node-health and degraded-share heatmaps, a run-status table, and — when the run
+ logs carry clock-offset diagnostics — an offset/drift CDF.
+ - Within-run time series: one figure per benchmark with a throughput-over-time and a
+ latency-over-time panel, plus a third panel holding the saturation curve when the run ramped its
+ offered rate. A run measured at a single rate has one saturation point per node, which the
+ cross-run throughput-vs-rate figure already shows, so that panel is left out. Up to six runs are
+ rendered, one per configuration, taking the configurations the per-node CDF grid selected,
+ alternating stream modes so both arms of a comparison are covered, and preferring the repetition its
+ panel shows. Selection depends only on the event stream, not on latency
+ data, so a throughput-only benchmark gets its trace.
+
+
+
The report states an experiment once and then never repeats it. Under the centered title sits a single line
+ naming the sweep, every dimension it measured with the values that dimension took, and the run count and
+ duration from the manifests. Each figure heading carries the categorical identity all its panels share —
+ the benchmark, and the stream mode when the sweep compared only one — and each legend entry names only
+ what tells its series apart from the figure's others, with short dimension tags (N15 P16384
+ R1000). A run-scoped figure is headed by its run's compact configuration label; the run base
+ itself appears in the figure's data note, which names the CSVs it read. Within a panel grid the y-axis
+ label is printed in the first column only and the x-axis label only where no panel sits below, so a row
+ of facets sharing one unit spends the reclaimed width on the data. A figure's legend is fitted to its
+ panel grid's width, laying out as many entries to a row as fit rather than breaking after a fixed count,
+ and consecutive per-node figures over the same nodes share the legend of the first.
+
+
Figures are grouped into sections, each starting a page: Scaling (metric against a swept
+ dimension, plus goodput and per-operation cost), Load curves (throughput-latency),
+ Stream-mode comparison (the ratio figures), Cluster health (the heatmaps, the
+ run-outcome table, and the clock-offset CDF), Per-node latency (the CDF grid), Within-run
+ time series, and Failed runs. Each section opens with a note stating what its figures show
+ and, where the report has to choose which runs to draw, how it chose them.
+
+
Figures are drawn natively at compile time by gorumsplot.typ, a cetz-plot helper library
+ embedded in the sweep binary and copied into each report directory, so the report recompiles
+ anywhere Typst is installed. Each figure appears only when its data supports it: a metric-vs-dimension
+ figure when that dimension varies, the ratio figures when exactly two stream modes are comparable, the
+ offset CDF when the logs carry offset lines, and so on.
+
+
Within-run time-series CSV columns
+
Per benchmark, the report writes <bench>_throughput.csv,
+ <bench>_latency.csv, and <bench>_saturation.csv with these columns:
+
+
+ | Plotter | Reads | CSV columns |
|---|
+
+
+ ThroughputTimePlotter | ThroughputInterval, PhaseMarker | offset_s, throughput_ops_s, phase, node |
+ LatencyTimePlotter | LatencyInterval, PhaseMarker | offset_s, mean_ns, stddev_ns, count, phase, node |
+ SaturationCurvePlotter | PhaseMarker(RATE_STEP), ThroughputInterval, LatencyInterval | offered_rate, throughput_ops_s, mean_latency_ns, node |
+
+
+
Multi-node input stays distinguishable: every row carries the node it came from (the
+ Report label, falling back to the filename stem), rows are grouped per node — each node's
+ series is contiguous, not globally sorted by offset — and the saturation curve keeps one set of rate
+ levels per node. The sweep's -trim flag drops interval events before that offset (phase
+ markers always pass through), consistent with the run summary.
+
+
Cross-run figures
+
The cross-run figures come from the rep-averaged agg.csv: sweep -plot discovers
+ runs through the per-run manifests (§13), reads the benchmark name and sweep
+ dimensions from each result's config, aggregates the per-node files of each run (throughput
+ summed across nodes, latency samples merged; hdr runs contribute their weighted histogram pairs instead),
+ applies the trim recorded in the manifest — the same interval/index-map cut as the run summary
+ (§10) — and averages repetitions into a mean with a sample SD and 95% CI, keeping
+ stream_mode as a series dimension when present. Each figure is produced only when its varying
+ dimension actually varies in the data. The figures and the sweep dimension each one needs:
+
+
+ | Figure | Shows | Requires variation in |
|---|
+
+
+ throughput_vs_{nodes,workers,payload,rate,send_buffer,recv_buffer} | Aggregate throughput vs a sweep dimension, faceted by the fewest-valued other dimension | that dimension |
+ latency_vs_{nodes,workers,payload,rate,send_buffer,recv_buffer} | Median (p50) latency vs a sweep dimension | that dimension |
+ goodput_vs_payload, goodput_vs_nodes | Cluster byte throughput (throughput × payload) — the sustained bandwidth in bytes/s | -payload (> 0) |
+ mem_per_op_vs_nodes, allocs_per_op_vs_nodes | Heap bytes/op and allocations/op vs N | -n |
+ tl_curve_{workers,rate} | Latency vs achieved throughput, one panel per N, one point per level of the load dimension the curve traces along; split into scale-band figures when peak latencies span a wide range. A sweep that varies both load dimensions gets one figure per dimension | -workers or -rate (per N) |
+ throughput_ratio_vs_{nodes,workers,payload,rate,send_buffer,recv_buffer}, latency_ratio_vs_{nodes,workers,payload,rate,send_buffer,recv_buffer} | Non-baseline/baseline ratio vs a sweep dimension, faceted by the fewest-valued other dimension, with a parity reference line | exactly two -stream-modes per configuration, that dimension varying within one comparable series, and — for the latency ratio — latency data in both modes |
+ per_node_cdf | One figure whose panels are per-node latency CDFs, three to a row and fifteen to a page, one panel per run. Runs are the first repetition of a configuration (the full dimension tuple, buffer capacities included) selected to spread across the sweep: a configuration earns a panel when it is the first to measure some dimension value, so the panels cover every node count, payload, rate, and mode rather than crowding into whichever corner sorts first. The remaining panels are filled in run order, so the page is used | nothing — produced for any run with CDF data |
+ node_health | Hosts × configurations heatmap of per-node throughput relative to each run's median; a slow host shows as a red cell, and a host absent from a configuration as a grey one. A color scale below the grid states the range | nothing — produced for any sweep |
+ degraded_share | Heatmap of the degraded-repetition fraction, node count and mode on one axis and the remaining varying dimensions on the other, so one configuration label is split over two axes instead of crowding one | any degraded repetition |
+ run_status | Table of run outcomes (succeeded/degraded/failed) per node count | any degraded or failed run |
+ clock_offsets | Empirical CDF of the absolute clock offset and residual drift, parsed from the run logs | offset lines in the logs |
+ failed_time_series_* | Under a “Failed runs” heading: the throughput and latency traces of failed runs, one representative per (configuration, error signature) group and at most three in total, each noting how the run failed and how many of its nodes wrote no result file at all. Whatever the cap drops is logged | a failed run with event data |
+
+
+
Three flags re-shape the report without re-running the sweep. -exclude DIM=VALUE
+ (comma-separated; dims: nodes, workers, payload, rate,
+ send_buffer, recv_buffer, stream_mode, benchmark)
+ drops matching data points — useful when one extreme
+ value stretches every axis and hides the differences among the rest, e.g. -exclude nodes=58
+ (a doubled-up node count). -exclude-run NAME drops a specific run by its base name.
+ The report also names, in its log, every repetition whose throughput differs from its configuration's
+ median by more than 1.4x in either direction — the analysis-side backstop behind the sweep's own
+ per-node bounds, for a directory collected before those bounds existed or with them disabled.
+ -include-degraded keeps degraded runs (a node outside the -degraded-below/-degraded-above bounds
+ of the run median) in the aggregate figures; by default they are excluded, which biases a mode's
+ aggregates upward when degradation clusters in that mode, so compare both variants before drawing
+ conclusions. The run-status and clock-offset diagnostics describe the whole directory and ignore these
+ per-configuration filters. When two or more stream modes are present, color encodes the mode and the line
+ dash is assigned from the remaining held-fixed dimensions only, so e.g. the dual and dedup lines of the
+ same payload read as one pair.
+
+
sweep -export-csv <dir> is a separate, standalone conversion, not a report filter: it
+ regenerates plotdata/runs.csv and plotdata/nodes.csv from that directory's
+ plotdata.binpb and exits without touching report/. Reach for it when inspecting
+ or scripting against the collected data directly — grepping for a run or a node — rather than for
+ plotting.
+
+
sweep -export-compact <work-dir> is the other standalone conversion, and the driver-side
+ half of a download: it rebuilds plotdata/ and compact-transfer/ from the raw
+ result files still in a finished run's work directory, then exits. Run it on the driver when a run's
+ compact transfer predates a change to what gets exported — the event streams, say — so the laptop can
+ download the rebuilt directory instead of the whole raw archive. Add -collect-profiles to
+ include the per-node profiles. Note that the driver's /tmp is not durable, so this works only
+ while the work directory survives.
+
+
+
15. Sweep recipes
+
+
Commands to exercise the benchmarks and produce each figure family. All assume the repository root, SSH host
+ aliases like bb1…bb30 in ~/.ssh/config (see
+ scripts/bbchain-ssh-config), and the tools built via make sweep and (for local
+ runs) make benchmark. Quote bracketed host ranges so the shell does not expand them. Compiling
+ the report to PDF needs Typst installed; without it the sweep still writes
+ the CSVs and report.typ.
+
+
Preflight and smoke test
+
Run the host diagnostics first. The second command is a quick smoke test on three nodes.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -check
+
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-3]' -test 3
+
+
+
Stream deduplication evaluation
+
These recipes compare the same benchmark binary in dual and dedup stream modes.
+ They use the cluster-local driver and detach the run.
+
Smoke recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -test 9 -sweep stream-dedup-smoke -stream-mode dual,dedup -benchmarks SymmetricQuorumCall,SymmetricMulticast
+
+
Balanced N-scale recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -sweep stream-dedup-nscale -n 5,9,15,21,27 -workers 1 -payload 0 -rate 0 -benchmarks SymmetricQuorumCall,SymmetricMulticast -stream-mode dual,dedup -duration 20s -trim 5s -interval 500ms -stats-mode hdr -reps 3
+
+
Offered-load recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -sweep stream-dedup-load -n 9,21,27 -workers 1,2,4,8,16,32 -payload 0 -rate 0 -benchmarks SymmetricQuorumCall -stream-mode dual,dedup -duration 20s -trim 5s -interval 500ms -stats-mode hdr -reps 3
+
+
Payload recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -sweep stream-dedup-payload -n 9,21 -workers 4 -payload 0,64,1024,16384 -rate 0 -benchmarks SymmetricQuorumCall,SymmetricMulticast -stream-mode dual,dedup -duration 20s -trim 5s -interval 500ms -stats-mode hdr -reps 3
+
+
Collect recipe:
+
+
+
./cmd/sweep/sweep -collect
+
+
Plot recipe:
+
+
+
./cmd/sweep/sweep -plot <collected-outdir>
+
+
+
Buffer sizing - throughput_vs_send_buffer, throughput_vs_recv_buffer
+
+
The send queue is the per-node queue of outbound requests
+ (-send-buffer, default 4096); the receive queue carries finished handler responses to the
+ goroutine that writes them back (-recv-buffer, default 0, unbuffered). Both are swept as
+ ordinary dimensions, so one detached invocation covers every capacity and the report plots throughput and
+ latency against them.
+
+
The receive queue only affects benchmarks whose handlers reply, so sweep it with two-way calls. The send
+ queue's depth is only exercised by a caller that pipelines, so include AsyncMulticast and set
+ -max-async above the largest capacity under test, or the async depth becomes the limiter
+ instead of the buffer.
+
+
Receive-side recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-8]' -driver first -detach -sweep recv-buffer -n 3,5,7 -workers 1,8,64 -recv-buffer 0,1,16,256 -benchmarks QuorumCall,SlowServer -duration 20s -trim 5s -reps 3
+
+
+
Send-side recipe:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-8]' -driver first -detach -sweep send-buffer -n 3,5,7 -workers 1,8 -send-buffer 64,256,1024,4096 -benchmarks Multicast,AsyncMulticast,SymmetricMulticast -duration 20s -trim 5s -reps 3 -extra-args '-max-async=8192'
+
+
+
Both sides at once, as a grid:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-8]' -driver first -detach -sweep buffers -n 5 -workers 8 -send-buffer 64,1024 -recv-buffer 0,16 -benchmarks QuorumCall,AsyncMulticast -duration 20s -trim 5s -reps 3 -extra-args '-max-async=8192'
+
+
+
Measurements on an 8-node LAN cluster found both curves flat: across
+ -recv-buffer 0,1,16,256 and -send-buffer 64,256,1024,4096, every difference in
+ throughput fell inside the 95% confidence interval. A queue only has to be deep enough that the goroutine
+ draining it never starves, which a saturating closed-loop client reaches at a small depth. Size these for
+ failure behavior rather than throughput: the send capacity is also the backlog at which a peer that stopped
+ draining fails two-way requests with ErrSendQueueFull, so it must stay above the number of
+ concurrent callers, and it is allocated eagerly at 64 bytes per slot per channel.
+
+
Three-variant comparison: baseline vs dual vs dedup
+
The dual series above measures the stream-dedup implementation with deduplication switched
+ off, which is not the same binary as the pre-dedup code. To separate the cost of the implementation from
+ the effect of the topology, add a baseline series built from a commit that predates stream
+ deduplication. A baseline sweep deploys that prebuilt binary via -binary and records
+ stream_mode=baseline in its manifests; the binary itself never sees the
+ -stream-mode flag. Baseline cannot be mixed with other modes in one invocation — one sweep
+ deploys exactly one binary — so it runs as a second sweep with the same dimensions. No branch switching
+ is needed while experiments run: the baseline binary is built once from a git worktree.
+
Build the baseline benchmark binary once, from the pre-dedup commit or branch you compare against:
+
+
+
git worktree add /tmp/gorums-baseline <pre-dedup-ref>
+GOOS=linux GOARCH=amd64 go -C /tmp/gorums-baseline build -o /tmp/benchmark-baseline ./cmd/benchmark
+git worktree remove /tmp/gorums-baseline
+
+
Run the feature sweep (dual,dedup) and the baseline sweep with identical dimensions:
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -sweep dedup-eval -n 5,9,15,21,27 -workers 1 -payload 0 -rate 0 -benchmarks SymmetricQuorumCall,SymmetricMulticast -stream-mode dual,dedup -duration 20s -trim 5s -interval 500ms -stats-mode hdr -reps 3
+./cmd/sweep/sweep -hosts 'bb[1-30]' -driver first -detach -sweep dedup-eval-baseline -n 5,9,15,21,27 -workers 1 -payload 0 -rate 0 -benchmarks SymmetricQuorumCall,SymmetricMulticast -stream-mode baseline -binary /tmp/benchmark-baseline -duration 20s -trim 5s -interval 500ms -stats-mode hdr -reps 3
+
+
After collecting both runs, merge their plotdata.binpb files into one folder; every figure then
+ shows baseline, dual, and dedup as separate series because
+ stream_mode is a series dimension. PlotData has only one repeated field
+ (runs), so concatenating two serialized messages and reading the result back as one message
+ merges their runs — no header handling needed, unlike the old CSV pair:
+
+
+
mkdir -p combined/plotdata
+cat out/dedup-eval/plotdata/plotdata.binpb out/dedup-eval-baseline/plotdata/plotdata.binpb > combined/plotdata/plotdata.binpb
+./cmd/sweep/sweep -plot combined
+
+
Interpretation: a gap between baseline and dual flags overhead introduced by the
+ dedup implementation itself (it should be near zero); the dual vs dedup gap
+ isolates the effect of halving the stream count.
+
+
Node scaling - throughput_vs_nodes, latency_vs_nodes, mem_per_op_vs_nodes, allocs_per_op_vs_nodes
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-25]' -sweep nscale \
+ -n 3,5,9,17,25 -workers 4 -duration 10s -trim 1s \
+ -benchmarks SymmetricQuorumCall,SymmetricMulticast
+
+
One node per host is ideal; with fewer hosts than the largest -n, sweep packs extra nodes onto
+ hosts with successive ports. The default -benchmarks is SymmetricQuorumCall
+ alone; the recipe adds SymmetricMulticast for both symmetric benchmarks, and
+ -benchmarks QuorumCall,Multicast exercises the coordinator-style benchmarks instead
+ (every node coordinates against all peers). Sweep treats each comma-separated entry as an exact benchmark
+ name and anchors it automatically - it is not a regexp.
+
+
Closed-loop runs with few workers measure latency scaling, not capacity
+
The recipe above uses -rate 0 (the default closed-loop mode): each worker fires its next op
+ immediately after the previous one completes. By Little's Law, aggregate throughput ≈ N × workers /
+ mean_latency. With a small fixed worker count (e.g. -workers 4), the
+ throughput_vs_nodes figure is therefore algebraically equivalent to the latency curve -
+ inverted and scaled by N - not a measurement of capacity.
+
Two structural artifacts follow. For SymmetricQuorumCall, the N=3 point is not comparable to
+ the rest of the series: a majority threshold of 2 out of 2 outbound peers means the call completes
+ after the fastest reply (the minimum order statistic), so p50 latency is far lower than at N=5 where the
+ 3rd of 4 replies must arrive. For SymmetricMulticast, the latency measures time in the
+ gorums send queue (SendTime is stamped before the message is handed to the gRPC
+ transport), so high latency at small N reflects a deep per-stream queue rather than network delay; at
+ large N (17/25) the workers become the bottleneck and the throughput drop reflects worker blocking on
+ backpressure rather than network saturation.
+
For a true throughput ceiling, see Getting a true throughput curve below, or use E3, E4, or E6
+ in the paper experiment plan (§16).
+
+
+
Concurrency scaling - throughput_vs_workers, latency_vs_workers
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep concurrency \
+ -n 9 -workers 1,2,4,8,16,32 -duration 10s -trim 1s
+
+
+
Throughput-latency curve - tl_curve_{workers,rate}
+
Needs variation in a load dimension — the worker count or the offered rate — per node count; sweeping the load
+ dimension and the node count yields one panel per N. A rate sweep at a fixed worker count gets the curve just
+ as a worker sweep does, traced along the rate instead.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-17]' -sweep tlcurve \
+ -n 5,9,17 -workers 1,2,4,8,16 -duration 10s -trim 1s
+
+
+
Payload scaling - throughput_vs_payload
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep payload \
+ -n 9 -payload 0,64,256,1024,4096,16384 -duration 10s -trim 1s
+
+
+
Paced (rate-limited) runs
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep rate \
+ -n 9 -rate 1000,2000,5000,10000,0 -benchmarks SymmetricMulticast \
+ -duration 10s -trim 1s
+
+
Rate 0 is the saturating reference point. sweep -plot treats rate as a sweep dimension:
+ with two or more non-zero rates it produces the throughput_vs_rate and
+ latency_vs_rate figures (rate 0 is excluded from the load axis), and in the other figures runs
+ differing in rate stay separate, with the legend annotating each series' rate.
+
+
Getting a true throughput curve
+
A node-scaling run at a fixed small worker count measures how latency changes with N, not how much work the
+ system can actually do. Three approaches give a honest capacity measurement.
+
Worker sweep at fixed N (recommended for a first look). Vary -workers while
+ holding N constant. Throughput rises with workers until the system saturates, then plateaus; the plateau is
+ the capacity. This is what E3 and E4 measure - combine with node scaling to see both dimensions.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep workers-n9 \
+ -n 9 -workers 1,2,4,8,16,32,64 -duration 10s -trim 1s \
+ -benchmarks 'SymmetricQuorumCall,SymmetricMulticast' -reps 3
+
+
Rate sweep (open-loop). Hold workers fixed at a level above the saturation point (for
+ example -workers 64) and vary -rate. sweep -plot produces
+ throughput_vs_rate and latency_vs_rate; the knee where delivered throughput
+ diverges from offered rate is the saturation point. The pacer fall-behind warning (§9)
+ tells you when workers are too few to sustain the requested rate.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep rate-n9 \
+ -n 9 -workers 64 \
+ -rate 5000,10000,20000,40000,80000,0 \
+ -benchmarks 'SymmetricQuorumCall,SymmetricMulticast' \
+ -duration 10s -trim 1s -reps 3
+
+
Rate ramp (single-run saturation curve). The rate-ramp flags (§7)
+ sweep offered load within one execution, tracing the complete saturation curve in a fraction of the wall
+ time. Use the saturation recipe below and size workers for the ceiling rate: sustaining R ops/s at
+ per-op latency L requires workers ≥ R × L (§9).
+
+
Error bands
+
Add -reps 3 (or more) to any recipe; sweep -plot averages repetitions and draws the
+ mean with a 95% CI band.
+
+
Rendering the report
+
A local sweep and a driver collection generate the report automatically. Regenerate it by hand — for example
+ to apply different filters — by pointing sweep -plot at the sweep directory; it writes the CSVs
+ and report.typ under <dir>/report and compiles report.pdf when
+ Typst is installed. Driver compact results are enough for the report; collect the retained raw
+ .binpb archive only when you need archival data or custom analysis.
+
+
+
./cmd/sweep/sweep -plot out/<label>
+
+
+
Saturation curve via rate ramping
+
The ramp flags pass through sweep, so a distributed run traces the curve in one execution; the ramp starts
+ at -rate-step and climbs to -rate-step-max, dividing -duration
+ evenly across the levels (§7). The same flags also work for a standalone local
+ benchmark run.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep saturation \
+ -n 9 -benchmarks SymmetricQuorumCall -duration 10s \
+ -rate-step 1000 -rate-step-max 10000
+
+
+
+
./cmd/sweep/sweep -plot out/saturation
+
+
The same ramp flags also work for a standalone local benchmark run, which writes one result file with the
+ rate-ramp event stream; build the local binary first.
+
+
+
make benchmark
+
+
+
+
./cmd/benchmark/benchmark -benchmarks QuorumCall -config-size 4 -time 10s \
+ -rate 1000 -rate-step 1000 -rate-step-max 10000 \
+ -output out/ramp.binpb
+
+
+
Profile collection and PGO
+
-collect-profiles downloads each node's CPU and heap profiles next to its result file;
+ -pgo additionally merges the CPU profiles into default.pgo
+ (§13). Profile a saturating workload - the hot paths under load are what PGO should
+ optimize.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep profile \
+ -n 9 -workers 8 -duration 30s -pgo
+
+
+
+
go tool pprof -top out/<label>/profile_*_bb1_9000.cpu.prof
+
+
+
+
go build -pgo out/<label>/default.pgo -o cmd/benchmark/benchmark ./cmd/benchmark
+
+
+
Fault injection: kill nodes mid-run
+
-fault-kill-after (§9) makes every node exit cleanly partway through;
+ the time series shows the cluster's throughput before and after the loss, and the summary reports the
+ surviving nodes. Passed via -extra-args it applies to all nodes, so use a duration shorter
+ than -duration and read the per-interval data rather than the aggregate.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-5]' -sweep faults \
+ -n 5 -benchmarks SymmetricQuorumCall -duration 15s \
+ -extra-args '-fault-kill-after=10s'
+
+
To kill only a subset of nodes, start the doomed node(s) by hand with the flag and let sweep run the rest,
+ or script the launch directly against the benchmark binary.
+
+
Foreign protocol binary
+
Any binary that implements the flag contract (§9) can be swept. Pass a prebuilt
+ linux/amd64 binary, or a build command with the {{output}} token.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep paxos \
+ -n 9 -binary ./paxos-linux-amd64 -benchmarks Commit \
+ -extra-args '-leader-timeout=500ms'
+
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep paxos \
+ -build 'go build -o {{output}} ./cmd/paxos' -n 9
+
+
+
Long runs with bounded memory
+
For multi-minute runs where the exact store's 8 B/op growth matters, switch the aggregate store to
+ hdr: constant memory with percentiles accurate to three significant figures
+ (§4). The distribution is whole-run; -trim still applies to the
+ throughput columns via the event stream.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep soak \
+ -n 9 -workers 4 -duration 5m -stats-mode hdr -interval 1s -trim 5s
+
+
+
LLM failure triage
+
Add -explain to a sweep to have a model diagnose its failed runs (§17).
+ The diagnosis is printed and written into each failed run's manifest.json as a
+ diagnosis field. With -driver the triage runs on the driver, which is the only
+ side of the firewall that can reach the UiS Ollama server; the diagnosed manifests then travel back with the
+ results. The API key comes from the environment, never a flag: on a plain run it must be set on the laptop,
+ and with -driver the laptop's OLLAMA_API_KEY is forwarded to the driver over the
+ SSH channel (never written to disk or the streamed log).
+
+
+
export OLLAMA_API_KEY=... # from Open WebUI: Settings → Account
+./cmd/sweep/sweep -hosts 'bb[1-25]' -driver first -sweep nscale \
+ -n 3,5,9,17,25 -workers 4 -explain -explain-model llama3.3
+
+
Because triage runs only after a sweep, a broken provider, model name, key, or endpoint would otherwise
+ surface only at the very end - too late to re-triage the failed runs cheaply. Two safeguards address this.
+ When -explain is set, the sweep runs a connectivity preflight before doing any work: it sends a
+ trivial prompt to the configured model and requires a non-empty reply. This runs where triage will run - on
+ the driver for a -driver sweep, the side that can reach the Ollama server - so an unreachable
+ endpoint, empty reply, or error page aborts the sweep at minute 0 instead of after the run. A missing key or
+ model is caught even earlier, in flag validation on the laptop, before anything is built or shipped.
+
For an on-demand check, -explain-check sends the same prompt and prints the reply and latency,
+ then exits. With -driver it ships the check to the driver - building and uploading just the
+ sweep binary, running the check there with the forwarded key, then removing the temp dir - so the laptop can
+ verify the firewalled UiS Ollama server it cannot reach itself. Without -driver it runs locally
+ against a provider the laptop can reach (OpenAI, Claude, or a local Ollama). Either way a missing key or
+ model fails fast, before anything is built or shipped.
+
+
+
# from the laptop, checks the UiS Ollama server on the driver:
+OLLAMA_API_KEY=... ./cmd/sweep/sweep -hosts 'bb[1-25]' -driver first -explain-check -explain-model llama3.3
+
+# locally against a provider the laptop can reach:
+ANTHROPIC_API_KEY=... ./cmd/sweep/sweep -explain-check -explain-provider claude -explain-model claude-opus-4-8
+
+
+
16. Paper experiment plan
+
Seven independent sweep runs covering the key performance dimensions of Gorums.
+ Each run is self-contained so it can be rerun in isolation if needed.
+ All runs assume 30 cluster nodes bb1…bb30 with SSH aliases configured as in
+ scripts/bbchain-ssh-config, tools built via make sweep, and
+ Typst installed to compile the report to PDF.
+
+
Before any run
+
Check cluster health and clock skew. Stale processes or large offsets invalidate latency measurements,
+ especially for server-measured benchmarks. A host with a lossy link contaminates every run it joins
+ (throughput understated, tail latency and CV inflated): the -netcheck preflight aborts on
+ hard loss, but exclude any host it or the degraded-run flagging names from -hosts until
+ its link is fixed (troubleshooting §5).
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-30]' -check
+
+
+
+
+
+
+ | Run |
+ Label |
+ Swept dimension |
+ Paper figures |
+
+
+
+ | E1 | e1-coord-nscale | nodes (coordinator model) | throughput_vs_nodes, latency_vs_nodes, mem_per_op_vs_nodes, allocs_per_op_vs_nodes |
+ | E2 | e2-sym-nscale | nodes (symmetric / P2P model) | throughput_vs_nodes, latency_vs_nodes, mem_per_op_vs_nodes, allocs_per_op_vs_nodes |
+ | E3 | e3-tlcurve | nodes × workers | tl_curve_workers (one panel per N) |
+ | E4 | e4-concurrency | workers (fixed n=9) | throughput_vs_workers, latency_vs_workers |
+ | E5 | e5-payload | payload size (fixed n=9) | throughput_vs_payload |
+ | E6 | e6-saturation | rate ramp (saturation curve) | <bench>_saturation.csv → the saturation panel of the time-series figure |
+ | E7 | e7-async | workers: sync vs async QC | throughput_vs_workers, latency_vs_workers |
+
+
+
+
E1 — Coordinator node scaling
+
How do QuorumCall and Multicast throughput, latency, and memory cost per
+ operation scale with quorum size? Uses the coordinator topology (one client against N servers).
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-25]' -sweep e1-coord-nscale \
+ -n 3,5,9,13,17,25 -workers 4 -duration 10s -trim 1s \
+ -benchmarks 'QuorumCall,Multicast' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e1-coord-nscale
+
+
Needs up to 25 nodes. The cost figure requires server memory stats, which are collected
+ automatically by StopRemote in client-measured runs. The -benchmarks flag is a
+ comma-separated list of names (each runs as a separate sweep cell); do not use | here - that
+ is the benchmark binary's own regexp syntax and will be interpreted as a shell pipe by the remote node.
+
+
E2 — Symmetric (peer-to-peer) node scaling
+
Same dimensions as E1 but for the symmetric topology, where every node is both client and server.
+ Directly comparable to E1: a higher coordinator-model throughput at the same N reflects the coordination
+ overhead removed by the symmetric design.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-25]' -sweep e2-sym-nscale \
+ -n 3,5,9,13,17,25 -workers 4 -duration 10s -trim 1s \
+ -benchmarks 'SymmetricQuorumCall,SymmetricMulticast' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e2-sym-nscale
+
+
At -workers 4 and -rate 0, the throughput_vs_nodes
+ figures reflect latency scaling under Little's Law (throughput ≈ N × 4 / mean_latency), not cluster
+ capacity. For SymmetricQuorumCall, interpret rising throughput at N≥5 as sub-linear latency
+ growth, not increased work capacity; the N=3 point is structurally incomparable (majority threshold 2/2
+ completes on the fastest reply). For SymmetricMulticast, the throughput drop at N≥17 reflects
+ workers blocking on gRPC backpressure as the fanout grows, not a network bottleneck; the one-way latency
+ (server-measured from SendTime) measures queue depth, not wire latency. E3 and E4 provide the
+ capacity measurements that E2 cannot.
+
+
E3 — Throughput-latency curve
+
Worker variation at several quorum sizes traces the achievable throughput-latency envelope. Each panel in the
+ figure shows one N-value; points within a panel are worker levels. Use SymmetricQuorumCall and
+ QuorumCall side-by-side to show both topologies in a single figure.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-17]' -sweep e3-tlcurve \
+ -n 5,9,17 -workers 1,2,4,8,16,32 -duration 10s -trim 1s \
+ -benchmarks 'SymmetricQuorumCall,QuorumCall' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e3-tlcurve
+
+
Needs up to 17 nodes. Workers must be sized for the worst-case latency at the highest
+ concurrency level (§9).
+
+
E4 — Concurrency scaling
+
How does each benchmark type respond to increasing in-flight concurrency at a fixed quorum size? Covers all
+ four benchmark variants so the figure shows relative concurrency headroom of sync/async and
+ coordinator/symmetric designs.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep e4-concurrency \
+ -n 9 -workers 1,2,4,8,16,32,64 -duration 10s -trim 1s \
+ -benchmarks 'QuorumCall,AsyncQuorumCall,Multicast,SymmetricQuorumCall,SymmetricMulticast' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e4-concurrency
+
+
+
E5 — Payload scaling
+
Message size impact on throughput. Shows where serialization and network bandwidth become the bottleneck.
+ Include all four main variants so the per-benchmark sensitivity to payload is visible in one figure.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep e5-payload \
+ -n 9 -workers 4 \
+ -payload 0,64,256,1024,4096,16384,65536 \
+ -duration 10s -trim 1s \
+ -benchmarks 'QuorumCall,AsyncQuorumCall,SymmetricQuorumCall,Multicast' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e5-payload
+
+
The 0-byte point is the framework overhead floor with no user data. The
+ 65536-byte point probes the network-bandwidth regime.
+
+
E6 — Saturation curve via rate ramping
+
A single run that sweeps offered load from a low baseline to a ceiling, tracing the throughput-latency
+ saturation curve without separate sweep configurations. The report draws it as the third panel of that run's
+ time-series figure, straight from the rate-ramp event stream; a run measured at one fixed rate has nothing to
+ trace there and gets two panels.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep e6-saturation \
+ -n 9 -workers 32 -duration 60s \
+ -benchmarks 'SymmetricQuorumCall,QuorumCall' \
+ -rate-step 500 -rate-step-max 15000
+
+
+
+
./cmd/sweep/sweep -plot out/e6-saturation
+
+
Workers must cover the worst-case latency at the ceiling rate: workers ≥ rate_max × L.
+ At 15 000 ops/s and ~1 ms latency, 32 workers provides headroom; revisit if the pacer fall-behind warning
+ fires (§9). The ramp has one level per 500 ops/s step up to the ceiling — 30 levels —
+ so 60 s gives ~2 s per level; increase -duration (or coarsen -rate-step) for
+ smoother curves.
+
+
E7 — Async vs synchronous QuorumCall
+
Direct comparison of QuorumCall (blocking) and AsyncQuorumCall (non-blocking,
+ completion-callback driven) under increasing concurrency. Highlights the latency and throughput advantage of
+ the async API when many operations are in flight.
+
+
+
./cmd/sweep/sweep -hosts 'bb[1-9]' -sweep e7-async \
+ -n 9 -workers 1,2,4,8,16,32,64 -duration 10s -trim 1s \
+ -benchmarks 'QuorumCall,AsyncQuorumCall' -reps 3
+
+
+
+
./cmd/sweep/sweep -plot out/e7-async
+
+
+
Rendering all runs
+
+
+
./cmd/sweep/sweep -plot out/<label>
+
+
The CSVs, report.typ, and (when Typst is installed) report.pdf land in
+ out/<label>/report/.
+
+
+
Reproducibility checklist
+
Before including any figure in the paper: (1) confirm sweep -check showed clock skew below 1
+ ms; (2) confirm no pacer fall-behind warnings in the sweep log; (3) confirm CV (σ/μ of interval
+ throughputs) is low (≲ 5%) - visible in the summary table's CV column and in the time-series plots; (4)
+ confirm -reps ≥ 3 error bands are tight; (5) confirm no manifest carries degraded_nodes
+ and the node-health heatmap shows no dark rows — a sick host contaminates every aggregate it joins.
+
+
+
+
+
+
17. Remaining tasks & roadmap
+
Work not yet implemented: deferred features that were analyzed in depth, stage-2/3
+ capabilities, and new proposals.
+
+
Depth & observability
+
+
Correctness verification stage 2
+
A workload verification token threaded through Control.Stop. The WithVerify
+ hook already exists in the harness (§8); wire a real agreement / state-hash
+ check for consensus workloads so a fast-but-wrong run is rejected rather than reported as a performance
+ result.
+
+
Broader fault injection stage 2
+
-fault-kill-after currently kills every node at time T (§9).
+ Extend it to a subset of nodes and to failure-detector tuning (kill-primary for view-change studies),
+ still in the fire-and-collect model — no out-of-band controller required.
+
+
Diagnostics follow-ups idea
+
Three gaps identified during cluster debugging, detailed in
+ troubleshooting §6: per-peer error detail when
+ EstimateOffsets fails (today only "k of N peers" is reported); time-scoped, severity-filtered
+ dmesg/journalctl capture in the failure snapshot before the kernel logs rotate
+ out; and a load-bearing variant of the -netcheck preflight, since the idle ping ring misses
+ links that only drop packets under connection churn.
+
+
Out-of-band control plane stage 3
+
A generic worker daemon driven by a central controller over a side channel — live metrics,
+ central timing, dynamic reconfiguration. Evaluate only if live metrics or central timing prove
+ necessary; it trades away the language-neutral binary contract that lets sweep drive foreign
+ binaries.
+
+
Client / server role split revisit
+
Distinct load-generating clients vs replicas, with independent counts and rates. Held for scheduling
+ reasons, not merit.
+
+
Packaging & format
+
+
Streaming event append deferred
+
The eventBuffer accumulates all events in a []*Event slice in memory until the run
+ completes, then serializes them into Result.events. For sub-10 ms intervals on multi-minute runs,
+ this can reach tens of megabytes:
+
+
+ | Interval | Duration | Events | Memory |
|---|
+
+
+ | 500 ms (default) | 10 s | ~42 | ~2 KB |
+ | 100 ms | 10 s | ~200 | ~10 KB |
+ | 10 ms | 10 s | ~2,000 | ~100 KB |
+ | 1 ms | 10 s | ~20,000 | ~1 MB |
+ | 1 ms | 5 min | ~600,000 | ~30 MB |
+ | 100 µs | 10 s | ~200,000 | ~10 MB |
+
+
+
A streaming append (e.g., a length-delimited event stream appended to the result file) would bound memory
+ to O(1) during the run. However, this requires framing changes: the current on-disk format is
+ [8-byte magic] + proto.Marshal(Report), with all events embedded in Report.events.
+ Streaming requires either a side file for events (two files per node; sweep would join them
+ when generating the report), or a format bump to v3 with length-delimited event records appended after
+ the Report (touching report.go, cmd/sweep/summary.go, and the sweep
+ report generator). Implementation cost is ~300–500 lines across multiple files, mostly format
+ plumbing.
+
Value is narrow: only performance engineers running high-resolution soak tests (sub-10 ms intervals,
+ multi-minute duration) benefit. The default 500 ms interval produces only hundreds of events (kilobytes).
+ The recommendation: defer until a concrete need arises. The least-disruptive implementation is a side file
+ (no magic bump, no format change), accepting the two-file-per-node cost.
+
+
Standalone benchkit module done
+
benchkit now lives in its own module, github.com/relab/gorums/benchkit, holding
+ the measurement library, the workloads built on it, and both commands. The dependency edge runs one way:
+ benchkit imports gorums, never the reverse, and the root go.mod carries no benchmarking or
+ orchestration dependencies.
+
This retired the reason sweep kept its own compiled-in copy of the result schema and its own
+ hand-synchronized forks of the read-time trim, the weighted-histogram statistics, and the time-series
+ renderer. Deleting the schema copy was not optional once sweep imported benchkit: both descriptor sets
+ register the same proto file name and message full names, so linking them into one binary panics at
+ init. The measurement forks were then verified equivalent and deleted, so no published
+ number moved; sweep now calls Summarize (§10),
+ DecodeReport (§12), LatencyDist, and
+ WriteTimeSeriesCSVs, keeping only the report-pipeline glue around them.
+
+
Protobuf result streaming & lint config idea
+
Evaluate streaming protobuf result messages instead of the protojson .json sibling
+ (§12), and add proto lint + formatting configuration for the schema files.
+
+
Orchestration & tooling
+
+
Failure-triage follow-ups idea
+
LLM failure triage (-explain) ships today: it diagnoses failed runs after a sweep and writes
+ the verdict back into each run's manifest.json (§13, recipe in
+ §15). Two extensions remain open: auto-triage of a single failed run mid-sweep
+ (before the remaining runs finish); and anomaly triage of successful runs flagged by their
+ statistics (high throughput CV, a pacer fall-behind warning, a single outlier node). The CV already
+ exists in the summary, so the latter is a natural follow-up.
+
+
Pluggable deployment backend deferred
+
sweep hard-wires SSH/iago as the way to launch nodes and collect files. The launcher role
+ — "start N nodes with this command; bring back these files" — is conceptually separable from
+ the statistics, the result schema, and the harness. A small launcher interface would let SSH stay the
+ default while a Docker/compose/k8s backend could be added later without touching the measurement code.
+ Kept as a seam to preserve, not a planned feature: Docker-based deployment is out of scope while the
+ cluster is SSH-reachable, and Go static binaries already give most of the reproducibility a container
+ would.
+
+
Broader gorums / project tasks
+
Tracked in benchmark/todo.md; the benchkit-adjacent ones:
+
+ - Move reusable code to
iago (e.g. SSH-config resolution), avoiding
+ moving code that depends on the copied resolveSSHConfigPath.
+ - CloudLab access — request a cluster and add its SSH config, as a second test
+ bed.
+ - Expose sender node ID via
ServerCtx instead of the benchmark-specific
+ TimedMsg.sender_id field; if adopted, that wire field can be removed.
+ - DNS-free
WithNodeList — configuration construction currently
+ resolves hostnames (normalizeAddr uses net.ResolveTCPAddr) and the
+ server-side path panics on a transient resolver failure; sweep works around it with numeric peer
+ addresses (§13). Preferred direction: make normalization syntactic
+ (literal hostnames, canonical numeric IPs), accepting weaker duplicate detection across aliases; an
+ explicit pre-resolved node-list option is the compatible fallback. A core API change to evaluate
+ separately.
+ - Exported-API review — unexport what need not be public; ensure naming, docs,
+ and examples are consistent across benchkit.
+ - PGO build wiring (the sweep already merges a
default.pgo; close the
+ loop by building the shipped benchmark binary with it) and httptrace-style gRPC call
+ tracing.
+
+
+
+
Reference documentation for the benchkit toolkit. Sections 1–14 describe the shipped
+ toolkit (the design rationale is §2); section 15 gives runnable recipes; section 16 is the paper
+ experiment plan; section 17 records remaining tasks and deferred features. Failure history and
+ log-analysis guidance live in benchkit-troubleshooting.html.
+
+